What is Pastebin?
Pastebin-like services let users paste plain text (source code, config files, logs) into a form, then receive a unique URL they can share with others. The receiver opens the URL and reads the pasted content without needing to sign up. Use cases: sharing code snippets, debugging logs, configuration files, and one-off data exchanges.
Similar to URL shortening, the core problem is mapping a short random key to content stored in a data store, and redirecting (or returning) the content on key lookup. The key differences: (1) we store potentially large text blobs, not just a URL, so we need separate object storage; (2) paste content can be up to 10 MB; (3) read/write ratio is 5:1, much lower than URL shortening's 100:1.
Requirements and Capacity Estimation
Functional requirements: (1) Users paste text and get a unique URL. (2) Accessing the URL returns the original text. (3) Pastes expire after a default timespan; users can set custom expiry. (4) Users can optionally set a custom alias. Non-functional: highly available, low-latency reads, paste links should not be guessable.
Capacity estimates: assume 1M new pastes/day (12 pastes/sec), 5M reads/day (58 reads/sec). Average paste = 10 KB → 10 GB of new data/day → 36 TB over 10 years. Key space: using Base64 with 6-character keys → 64^6 ≈ 68.7B unique keys (far more than the ~3.6B pastes in 10 years). Cache: 20% of daily reads = 0.2 × 5M × 10 KB ≈ 10 GB.
System APIs
REST API surface: • createPaste(api_dev_key, paste_data, custom_url?, user_name?, paste_name?, expire_date?) → URL string on success, error code on failure. • getPaste(api_paste_key) → paste text. • deletePaste(api_dev_key, api_paste_key) → true/false.
Throttle by api_dev_key to prevent abuse: limit each key to a maximum number of paste creations per time window. Reject pastes larger than 10 MB.
Database Design
Two tables are needed:
Paste table (metadata): URLHash (PK, varchar 6), ContentKey (object storage path), ExpirationDate, UserID, CreationDate. ContentKey is the S3/object-store key where the actual text lives.
User table: UserID (PK), Name, Email, CreationDate, LastLogin.
Since there are no relationships between records except user ownership, and we need to store billions of rows, NoSQL (like Cassandra or DynamoDB) works well for the Paste table. The user table can stay in MySQL. Shard the Paste table by URLHash.
Object storage (Amazon S3 or equivalent): store the actual paste text as a file at ContentKey. This separates the hot-path metadata lookup (small row from NoSQL) from the large text retrieval (object store fetch), and lets each scale independently.
graph TD Client --> LB["Load Balancer"] LB --> AppServer["App Servers"] AppServer --> KGS["Key Generation Service"] KGS --> KeyDB[(Key DB unused / used)] AppServer --> MetaDB[(Metadata DB NoSQL — URLHash → ContentKey)] AppServer --> ObjectStore["Object Store (S3) Actual paste text"] AppServer --> Cache["LRU Cache (Memcached/Redis)"] Cache --> MetaDB
High-Level Design and Component Design
Application layer handles all read/write requests:
Write flow: App server receives paste text → calls KGS for a unique 6-char key (moves key from unused_keys to used_keys atomically) → stores text in object storage at that key path → inserts metadata row (URLHash=key, ContentKey=S3 path, expiry) into NoSQL DB → returns short URL to user.
Custom aliases: if the user provides a custom key, check it against the DB first; return error if already taken.
Read flow: App server receives paste key → checks LRU cache for (key → S3 path); on hit, fetch text from object store → on miss, query NoSQL DB for ContentKey, update cache, fetch text → return text to user.
KGS single point of failure: solve with a standby replica that takes over on primary failure. App servers can also pre-cache a small batch of keys from KGS to avoid a network round-trip on every write.
Purging, Partitioning, and Caching
DB Cleanup: A lightweight background service periodically scans for pastes past their expiration date and deletes both the metadata row and the object-store file. Deleted keys are returned to KGS's unused_keys pool. Lazy deletion is also acceptable: check expiry on read and return 404.
Data Partitioning: Shard the metadata NoSQL DB using consistent hashing on URLHash. This distributes pastes evenly across shards and minimizes resharding overhead when adding nodes.
Cache: Store the 20% hottest pastes (key → S3 path, or key → text for small pastes) in Memcached with LRU eviction. 10 GB of cache covers 20% of daily read traffic.
Load Balancer: Place an LB between clients and app servers, and between app servers and DB/object-store. Use round-robin initially; switch to least-connections for uneven paste sizes.
Security: Pastes can be public or private (password-protected). Store a permission level and optional hashed password in the metadata row. Private pastes are not indexable by search engines.