Requirements and Scale
Functional: upload videos, stream videos, search, recommendations, like/dislike/comment, subscribe to channels. Non-functional: highly reliable (video never lost), smooth streaming (adaptive bitrate), reads dominate heavily.
Scale: 1.5B users, 30M daily active creators, 500 hours of video uploaded per minute, 1B hours watched per day. Average video = 5 minutes = 300 MB (1080p). 500 hrs × 60 × 300MB = 9 TB of raw video per minute uploaded.
Video Upload and Transcoding Pipeline
Raw upload: client uploads to an Upload Service which writes video to a raw blob store (S3). Upload completion triggers a message on a queue. Transcoding workers pick up the job and produce multiple resolution variants (360p, 480p, 720p, 1080p, 4K) plus audio tracks and subtitles. This is a CPU-intensive fan-out — typically 50+ worker processes per video. Each output variant is stored in S3 with a separate path.
Transcoding at YouTube scale uses Reed-Solomon encoding for fault-tolerant replication across storage nodes: encode data into M+N chunks where the original can be reconstructed from any M chunks. This allows N node failures without data loss.
Video deduplication: before storing a new video, compute a perceptual hash (using algorithms like Block Matching or Phase Correlation). If a matching video exists (pirated content, re-uploads), block or attribute the upload rather than storing a duplicate.
graph LR Creator --> UploadSvc["Upload Service"] UploadSvc --> RawStore["Raw Video (S3)"] RawStore --> TranscodeQueue["Transcoding Queue (Kafka)"] TranscodeQueue --> Workers["Transcoding Workers (360p/480p/720p/1080p)"] Workers --> CdnStore["Processed Video (S3 + CDN)"] CdnStore --> Viewer
Video Storage — BLOB Storage
Videos are Binary Large Objects (BLOBs). They cannot go in a relational DB — they use dedicated object storage (S3, GCS). Metadata (video_id, channel_id, title, description, duration, like_count, view_count, created_at, status) is stored in MySQL sharded by video_id.
Each video is split into segments (a few seconds each) and stored as individual objects. The video player downloads segments ahead of time (pre-fetching) and switches resolution based on available bandwidth — this is Adaptive Bitrate Streaming (ABR) using MPEG-DASH or HLS.
CDN Architecture
A global CDN (like Akamai or YouTube's own) caches video segments at edge nodes worldwide. When a viewer requests a video segment, the CDN checks its local cache; on a miss, it fetches from the origin S3 bucket and caches it for future viewers in that region. Popular videos are fully cached at the edge and are served without touching origin servers. CDN is 95%+ of YouTube's delivery traffic.
CDN nodes are tiered: edge nodes (POPs) → regional aggregation caches → origin (S3). Cache TTLs are set to days for processed video (immutable) and minutes for metadata (mutable).
Video Recommendations
Collaborative filtering: "users who watched video A also watched video B." Matrix factorization on user-video watch history. Deep learning models (YouTube's DNN) take hundreds of features as input: watch history, search history, demographics, context. Two-stage: (1) Candidate generation (millions → hundreds using ANN lookup), (2) Ranking (hundreds → 10-20 using a complex model). Recommendations are pre-computed in batch jobs and served from a serving layer cache.
Analytics — View Counts and Engagement
View counts cannot be incremented in MySQL on every view — too many writes. Instead, increment a counter in Redis (atomic INCR), then periodically batch-flush to the DB. Like/dislike counts work the same way. Comments are stored in a separate sharded DB (comment_id, video_id, user_id, content, timestamp). Trending videos use a stream processing job (Flink) that counts views per video per time window and surfaces the top-K.