Requirements and Scale
Functional: one-on-one text messaging, group chats (up to 500 members), online/last-seen status, read receipts, push notifications for offline users, message history stored permanently. Non-functional: very low latency (< 100ms message delivery), high consistency for message ordering.
Scale: 500M daily active users, 100B messages per day ≈ 1M messages/sec. Each message ~100 bytes → 10 TB/day of new message data.
High-Level Design
Users connect to a Chat Server (WebSocket) for bidirectional real-time communication. Each Chat Server handles tens of thousands of persistent connections. A load balancer with sticky sessions routes each user to the same Chat Server for their session lifetime. For message delivery: sender → Chat Server A → Message Queue → Chat Server B → recipient.
If recipient is offline, Chat Server stores the message and triggers a push notification (APNs for iOS, FCM for Android). When the user comes online, undelivered messages are fetched from the Message Store.
graph LR UserA["User A"] -->|WebSocket| CS_A["Chat Server A"] CS_A --> MQ["Message Queue (Kafka)"] MQ --> CS_B["Chat Server B"] CS_B -->|WebSocket| UserB["User B"] CS_A --> MsgStore[(Message Store HBase)] CS_A --> PushSvc["Push Notification Service"]
Message Storage — Why HBase?
Requirements for the message store: (1) Very high write throughput (millions of messages/sec), (2) Time-ordered reads per conversation, (3) Efficient fetching of a conversation's last N messages, (4) Low-latency random access by message ID.
HBase (wide-column NoSQL) fits perfectly: row key = thread_id + timestamp (gives natural time ordering), columns = message fields. HBase is optimized for high write rates with sequential disk access via LSM trees. Comparison to alternatives: MySQL can't scale to this write volume without massive sharding complexity. Cassandra also works but HBase is preferred at Facebook's scale for time-series message data.
Message ID: needs to be unique, sortable by time within a conversation. Use a dedicated sequence number generator per thread, or embed timestamp in the high bits of the ID (similar to Twitter Snowflake).
Group Messaging
For a group with N members, delivering one message requires sending to N-1 members. At 500-member groups, a single message triggers 499 deliveries. The chat server fans out to a Message Queue per group. Worker services consume from the queue and push to each member's Chat Server.
A separate Group Membership Service stores the member list for each group. On each message, fetch the member list, determine which members are online (check presence service), deliver to online members via WebSocket, and enqueue for offline members.
Online Presence and Read Receipts
Presence Service: tracks who is online using heartbeats. Client sends a heartbeat every 5 seconds while active. If no heartbeat for 30 seconds, mark as offline. Online status is stored in Redis (key: user_id, value: last_heartbeat_timestamp). Publishing status changes to friends is done via pub/sub — each user subscribes to their friends' status channels.
Read receipts: when a user opens a conversation, their client sends a "read up to message_id X" event to the server, which stores this in the DB and fans out the receipt to the sender's Chat Server, which delivers it via WebSocket.
Caching and Performance
Cache the last 15 messages of each active conversation in Redis. Most users only read recent messages, so this cache handles the majority of read traffic. For older messages, query HBase directly. Cache the friend/contact list and group membership lists. CDN is used for media (images, videos, audio messages) sent in chats — similar to Instagram's approach where the chat DB stores a CDN URL, not the raw bytes.