Latency vs Throughput: Two Different Performance Questions
Latency measures how long one operation takes. Throughput measures how many operations a system completes in a period. A design can have excellent throughput and terrible latency, or low latency and limited throughput. Treating the terms as interchangeable leads to weak capacity plans and confusing interview answers.
A supermarket makes the difference clear. Customer latency is the time from joining a line to finishing checkout. Throughput is the number of customers all checkout lanes complete per minute. Opening more lanes may increase throughput, but one customer with a complicated return can still experience high latency.
How Do You Measure Latency?
Latency is normally measured in milliseconds or seconds from a defined start to a defined finish. Always name the boundary: client-to-client response, server processing time, database query time, or queue wait plus processing.
Averages hide painful tails. If 99 requests finish in 20 ms and one takes 5 seconds, the average may look acceptable while one percent of users have a terrible experience. Use percentiles:
- p50: half of requests finish at or below this latency.
- p95: 95 percent finish at or below this latency.
- p99: 99 percent finish at or below this latency.
Tail latency grows when requests wait for locks, garbage collection, disk, network retries, overloaded dependencies, or the slowest of many parallel calls. A page that calls twenty services may be limited by the slowest critical response rather than the average service time.
For a foundation, review latency in System Design.
How Do You Measure Throughput?
Throughput uses units such as requests per second, transactions per second, messages per second, jobs per minute, or bytes per second. State whether the number is offered load, accepted load, or successfully completed work. A queue can accept 100,000 messages per second while consumers complete only 40,000; the missing difference becomes growing lag.
Measure reads and writes separately because they use different paths and resources. A database may serve 50,000 cached reads per second but only 2,000 durable writes per second. A video service may handle modest request QPS while network throughput dominates because each response is large.
The guide to throughput and System Design scale covers capacity, peak factors, and bottlenecks in more detail.
Why Can High Throughput Increase Latency?
As utilization approaches the capacity of a constrained resource, requests spend more time waiting. CPU run queues grow, database connections fill, locks contend, disks queue I/O, and network buffers accumulate packets. The system may continue completing many requests per second while individual response times become unpredictable.
This is why a service should not operate permanently at 100 percent utilization. Headroom absorbs bursts, node failures, deployments, and uneven load. Autoscaling based only on average CPU can react too late if queue depth or p99 latency is already rising.
Queueing and Little's Law
Little's Law gives a useful relationship for a stable system:
concurrency = throughput × time in system
If a service completes 1,000 requests per second and each request spends 200 ms in the system, average concurrency is about 200 requests. If latency rises to one second at the same throughput, concurrency grows to 1,000, consuming more connections and memory and increasing the chance of collapse.
How Batching Trades Latency for Throughput
Batching groups work so fixed overhead is paid once. A database can write one transaction containing many rows; a message consumer can fetch several records; a GPU can process a batch of inputs. This usually increases throughput but makes the first item wait for the batch to fill.
Choose a maximum batch size and maximum wait time. For an analytics pipeline, waiting 500 ms may be harmless. For search autocomplete, the entire user-visible budget may be under 100 ms, so large batches are inappropriate.
Compression creates a similar trade-off. More CPU time can reduce network bytes and improve end-to-end latency on slow links, but aggressive compression may hurt latency when CPU is the bottleneck. Measure the complete path.
How Caches Affect Latency and Throughput
A cache can improve both metrics: hits return quickly and remove work from the database, allowing the system to handle more requests. The benefit depends on hit rate, object size, and miss behavior.
Cache failures can reverse the gain. When many popular keys expire together, requests fall through to storage, latency spikes, and database throughput is exhausted. Use expiration jitter, request coalescing, stale-while-revalidate, admission policies, and hot-key replication when the workload requires them. The caching strategies guide explains these failure modes.
Queues, Backpressure, and Load Shedding
A queue smooths bursts and protects downstream services, but it does not create capacity. If producers continuously generate more work than consumers finish, lag grows without bound. Users may receive a fast “accepted” response while the actual job completes hours later.
Backpressure slows producers when consumers are saturated. Techniques include bounded queues, concurrency limits, rate limits, reduced fetch size, and explicit retry-after responses. Load shedding rejects low-priority work so critical requests retain acceptable latency.
Define a queue-lag objective, not only a message-ingestion rate. For a notification system, “99 percent of urgent notifications sent within 30 seconds” is more meaningful than “the broker accepts 50,000 messages per second.”
Want to master this with video lessons and real case studies? This topic is covered in depth in my Udemy course System Design Fundamentals for Interviews — 5.5 hours, rated 4.8★, built from real interview questions.
Latency vs Throughput Examples
Search typeahead
Latency dominates because every keystroke waits for suggestions. Precompute popular prefixes, cache top results near users, cancel obsolete requests, and keep the dependency graph small. See the Google Typeahead design for a complete example.
Video transcoding
Throughput and cost dominate for background jobs. The user does not require each video to finish in milliseconds, but the fleet must keep up with daily uploads. Batching, queues, priority tiers, and elastic workers are appropriate.
Payments
Both matter, but correctness constrains optimization. The checkout should respond within a clear latency target and the service must handle peak transaction volume without weakening idempotency, durability, or fraud checks.
Analytics ingestion
High write throughput matters, while seconds of processing latency may be acceptable. Partition events, batch durable writes, compress payloads, and monitor consumer lag. Interactive dashboards create a separate low-latency read workload.
How to Design for Both
- Set separate latency and throughput objectives for each critical operation.
- Estimate average and peak load, not just daily totals.
- Find the constrained resource with utilization, queue, and saturation metrics.
- Keep headroom for bursts and failures.
- Bound concurrency so overload does not cascade.
- Cache repeated work and batch non-interactive work.
- Use backpressure and load shedding before queues become unbounded.
- Measure p50, p95, p99, successful throughput, and queue lag together.
Performance testing should increase offered load gradually and observe where latency bends upward. The knee of the curve is often a safer capacity boundary than the point where requests finally fail.
How to Talk About Latency vs Throughput in 30 Seconds
“Latency is the time one request spends in the system, while throughput is the amount of work completed per unit of time. I set separate targets and measure latency with percentiles because averages hide slow tails. As utilization approaches a bottleneck, queueing raises latency even if throughput remains high. I use headroom, caching, bounded concurrency, batching for non-interactive work, and backpressure or load shedding during overload. I monitor successful throughput, p99 latency, saturation, and queue lag together.”
Common mistakes
- Using average latency without percentiles.
- Calling accepted queue writes completed throughput.
- Assuming more concurrency always increases throughput.
- Adding an unbounded queue instead of handling overload.
- Optimizing a component metric while ignoring end-to-end latency.
Key Takeaways
- Latency measures time per operation; throughput measures completed work per time.
- High utilization creates queues and long tail latency.
- Batching improves throughput by adding waiting latency.
- Caches may improve both metrics, but misses and stampedes can overload storage.
- Backpressure, bounded concurrency, and load shedding keep overload controlled.
Next Steps
Use the complete interview-preparation guide to place latency and throughput targets inside requirements, estimates, and architecture decisions rather than treating them as isolated definitions.