Requirements and Scale
Functional: search tweets by keywords, hashtags, users, date range. Results sorted by relevance and recency. Non-functional: freshness (tweets searchable within seconds of posting), low latency (< 500ms), high availability.
Scale: 1.5B users, 500M tweets/day (6,000 tweets/sec), 800M searches/day (10,000 searches/sec). Average tweet is 140 characters ≈ 300 bytes. 500M × 300B = 150 GB of tweet text per day.
Inverted Index — Core of Search
An inverted index maps each word to the list of tweet_ids that contain that word: {"love": [tweet_1, tweet_4, tweet_9], "twitter": [tweet_2, tweet_4]}. Each entry in the list is a posting: {tweet_id, score, timestamp}. The score accounts for recency and engagement (retweets, likes).
To search for "love twitter": intersect the posting lists for "love" and "twitter", score and rank results, return top-K. The intersection is fast because posting lists are sorted by tweet_id.
graph LR TweetStream["Tweet Stream (Kafka)"] --> Indexer["Real-Time Indexer"] Indexer --> InvertedIndex["Inverted Index (Distributed)"] SearchQuery --> SearchService["Search Service"] SearchService --> InvertedIndex SearchService --> Results["Ranked Results"]
Distributed Indexing
The index is partitioned across multiple servers. Two strategies: (1) Shard by word (term-based): all posting lists for words starting with "a-m" on server 1. Pro: simple fan-in; Con: hot words cause hotspots. (2) Shard by tweet_id (document-based): each server indexes all words from its subset of tweets. Pro: even load; Con: every query fans out to all shards and merges results.
Twitter uses document-based (tweet-based) partitioning. Each query is broadcast to all index shards, each shard returns its top-K matches, and a merge service selects the global top-K. This is called a scatter-gather pattern.
Dynamic Indexing for Freshness
Static index: built offline by batch processing historical tweets. Handles old tweets. Dynamic index: an in-memory index that receives new tweets in real-time via Kafka. New tweets are indexed within milliseconds. Search hits both indexes: merge results from dynamic (new) and static (historical) indexes.
The dynamic index is small enough to fit in RAM. It is periodically merged into the static index (daily). This two-tier index architecture ensures freshness without sacrificing the performance of the large historical index.
Ranking and Scoring
A search result is ranked by a score that combines: (1) Text relevance (BM25 or TF-IDF for keyword matching), (2) Recency (exponential decay — newer tweets score higher), (3) Engagement (retweet count, like count, reply count), (4) Social graph signal (tweets from people you follow rank higher), (5) Language and location match. Machine learning models (LambdaRank, XGBoost) trained on click data learn the optimal weight for each signal.
Caching and Replication
Popular searches ("World Cup", "election") repeat frequently. Cache the result sets in Redis with a short TTL (1-5 minutes) since Twitter search results change rapidly. The index shards have replicas for fault tolerance and read scaling. Write (indexing) goes to all replicas synchronously to avoid stale reads. If a shard is temporarily unavailable, searches degrade gracefully by querying remaining shards.