Requirements
Core features: a user can publish a post and see friends' posts in a reverse-chronological news feed. Feed contains text, images, and videos. A user can have up to 5,000 friends. System supports 10 million daily active users.
Two sub-problems to design: (1) Feed publishing — when a user creates a post, how does it appear in the feeds of all their friends? (2) Feed retrieval — when a user opens the app, how is their feed assembled and delivered quickly?
Feed Publishing: Fanout on Write (Push Model)
When a user creates a post, immediately compute and push the post into the news feed cache of every friend. The news feed cache stores a list of (post_id, user_id) pairs for each user, ordered by timestamp.
Pros: reading the news feed is fast — the cache is pre-computed and ready. No computation at read time.
Cons: if a user has 5,000 friends, publishing one post triggers 5,000 cache write operations. For a user with millions of followers (a celebrity, an influencer), publishing one post creates millions of writes simultaneously — a "hotkey" or fan-out storm problem that can overwhelm your systems. Wasteful for inactive users — computing and storing feed updates for users who have not logged in for months serves no one.
Feed Publishing: Fanout on Read (Pull Model)
Do not pre-compute feeds at write time. When a user requests their news feed, fetch the user's friend list in real time, query each friend's recent posts, merge them, sort by timestamp, and return the result.
Pros: no wasted computation for inactive users. No fanout storm for celebrities — a celebrity's posts are simply stored, and followers pull them on demand.
Cons: reading the news feed is slow — multiple database queries at request time. For a user with 5,000 friends, each feed refresh requires fetching thousands of posts from potentially thousands of shards.
Hybrid Approach
The best production systems combine both models. For most users (few friends, reasonable follower counts): fanout on write. Pre-compute their friends' feeds so reads are fast. For celebrities and users with massive follower counts: fanout on read. Do not attempt to push to millions of feeds instantly — let followers pull on demand.
The system identifies high-follower-count accounts (e.g., > 10,000 followers) and excludes them from the push fanout. When a regular user reads their news feed, the pre-computed feed (from push) is merged with any recent posts from celebrity accounts they follow (fetched on-demand via pull).
Fanout Service Architecture
The fanout service runs asynchronously, triggered by each new post:
1. Fetch the poster's friend IDs from a graph database (Neo4j, or a relational table with indexed friend relationships). 2. Filter based on user settings — if friend B has muted the poster, exclude B. If the post is restricted to a list, include only users on that list. 3. Put (friend_list, post_id) into a message queue. 4. Fanout workers consume from the queue and write (post_id, user_id) pairs to the news feed cache for each friend.
The news feed cache stores only IDs, not full post content. This keeps the cache compact — even with thousands of feed entries per user, the storage is minimal. Full post content (text, images, user profile data) is fetched separately from post and user caches when rendering the feed.
News Feed Retrieval
When a user opens the app and requests their news feed:
1. The load balancer routes the request to a web server. 2. The web server calls the news feed service. 3. The news feed service fetches the ordered list of post IDs from the user's news feed cache. 4. For each post ID, fetch full post content from the post cache (or post database on cache miss). Fetch author profile from the user cache. Merge into a complete feed item. 5. Return the assembled feed as JSON.
Media content (images, videos) is served from CDN — the feed response includes CDN URLs for images, not the raw bytes. The client fetches images from CDN directly.
Cache Architecture
Five layers of caching for the news feed system:
News Feed cache: pre-computed list of (post_id, user_id) pairs per user. Updated by the fanout service. Served directly on feed reads.
Post cache: full post content indexed by post_id. Populated when a post is created. Long TTL since posts rarely change after creation.
User cache: user profile data (username, avatar URL, bio) indexed by user_id. Medium TTL.
Social graph cache: user's friend IDs. Updated when friendships are created or removed. Used by the fanout service.
Counter cache: counts for likes, comments, and shares per post. These change frequently with high read volume — keeping them in Redis prevents constant database updates. Periodically synced to the database.