Definitions
Latency is the time elapsed between initiating an action and receiving the result. In a web service, latency is the time from when a client sends a request to when it receives the complete response. It is measured in milliseconds (ms) or microseconds (µs) and expressed as percentiles: p50 (median), p95, p99, p999.
Throughput is the number of operations completed per unit of time — requests per second (RPS), transactions per second (TPS), or messages per second. It measures how much work the system can sustain under steady load.
The goal is to achieve maximum throughput at acceptable latency. These two metrics are related but distinct, and optimizing one can hurt the other.
The Latency-Throughput Tradeoff
At low load, latency and throughput are independent — you can add more concurrent requests without affecting individual request latency. As load increases and the system approaches its capacity, queuing begins. Each new request waits for in-flight requests to complete, which drives up latency. Once latency starts rising, throughput levels off — you have hit the system's saturation point.
Little's Law quantifies this relationship: L = λW, where L is the average number of requests in the system, λ is the arrival rate (throughput), and W is the average time a request spends in the system (latency). This means if you want to sustain high throughput without degrading latency, you must reduce the time each request spends in the system — through faster processing, more parallelism, or better resource utilization.
Optimizing Latency
High latency comes from: slow database queries, synchronous network calls in the critical path, large serialized payloads, CPU-bound computation blocking the request thread, and lock contention.
Strategies to reduce latency: • Cache hot data in memory (Redis, Memcached) to avoid round-trips to the database. • Push slow operations off the critical path with async processing — return immediately and complete work in the background. • Use connection pooling to avoid TCP handshake overhead on every request. • Co-locate services that communicate frequently (same data center, same availability zone) to reduce network RTT. • Use binary protocols (Protobuf, Thrift) instead of verbose text (JSON) when the serialization overhead matters. • Move static assets to a CDN so the client fetches them from a nearby edge node rather than your origin server.
Optimizing Throughput
Throughput is limited by the slowest shared resource — the bottleneck. Identifying and eliminating bottlenecks is the discipline of capacity planning and performance engineering.
Common bottlenecks and fixes: • Single database writer → shard writes across multiple database nodes or use a write-ahead log with async replication. • CPU-bound single server → add more application server instances behind a load balancer; ensure servers are stateless so any node can handle any request. • Network bandwidth → compress responses, use HTTP/2 multiplexing, reduce payload size. • Lock contention → replace synchronized global structures with lock-free data structures or partition state so threads rarely contend.
Always benchmark under realistic load before and after optimization. A 10% latency improvement that reduces maximum throughput by 30% is not a net win.
Practical Reference: Latency Numbers
These rough numbers help you reason about system design trade-offs. Commit them to memory.
L1 cache reference: ~0.5 ns L2 cache reference: ~7 ns Main memory reference: ~100 ns SSD random read (4 KB): ~150 µs Network round-trip within same data center: ~500 µs Read 1 MB sequentially from SSD: ~1 ms HDD seek: ~10 ms Network round-trip across continents: ~150 ms
Key takeaways: Memory is 200× faster than a disk seek. An SSD random read is 30× faster than an HDD seek. A cross-continental round-trip is 300× slower than an in-datacenter round-trip. Design your hot paths to hit memory (cache) not disk, and keep latency-sensitive services in the same data center.