One client took down the API for everyone
A customer ships a bug: a retry loop with no backoff. Their integration starts calling your endpoint 4,000 times a second. Your database saturates. Every other customer starts seeing timeouts.
Nobody attacked you. There is no malice and no security hole. One badly written while loop in someone else's code took down your product for everybody else, and your system had no way to say no.
Rate limiting is how a system says no — deliberately, cheaply, and before the damage reaches anything expensive.
The analogy: the nightclub door
A club holds 200 people. The bouncer is not there because guests are bad people; the bouncer is there because the building has a capacity, and exceeding it hurts everyone inside.
Crucially, the bouncer stands at the door. Checking capacity after people are already on the dance floor is useless. Rate limiting works the same way: enforce it at the edge, before the request touches your application or your database.
The four algorithms
Fixed window
Count requests per client per fixed interval. 100 per minute: the counter resets at the top of every minute.
Simple and cheap — one counter with a TTL. It has one well-known flaw: a client can send 100 requests at 11:59:59 and another 100 at 12:00:00, putting 200 through in one second while never breaking the stated limit. That is the boundary burst problem.
Sliding window log
Store a timestamp for every request. To decide, count the timestamps in the last 60 seconds. Perfectly accurate, no boundary problem — and it stores one entry per request, which at scale is a lot of memory for a component whose whole job is to be cheap.
Sliding window counter
The practical compromise. Keep the current and previous window counters and weight the previous one by how far into the current window you are. Roughly: previous × (1 − elapsed_fraction) + current.
Two integers per client, no boundary burst, accuracy that is approximate but good enough. This is what most production limiters actually use.
Token bucket
A bucket holds N tokens and refills at a steady rate. Each request removes one token; empty bucket means reject.
The property that makes it the usual default: it allows bursts up to the bucket size while capping the long-run average. A user who has been idle for a minute has a full bucket and can fire 100 requests instantly — which is exactly what a real user does when they open an app that loads ten resources at once. A pure average-rate limiter would reject that legitimate burst.
Leaky bucket
Requests queue and drain at a constant rate. Output is perfectly smooth, which is ideal when you are protecting something that cannot tolerate variance. The cost is added latency, since requests wait in the bucket rather than being answered or rejected immediately.
Want to master this with video lessons and real case studies? This topic is covered in depth in my Udemy course System Design Fundamentals for Interviews — 5.5 hours, rated 4.8★, built from real interview questions.
What do you limit by?
The key you count against decides whether the limiter helps or hurts:
- API key or user ID — the right default for authenticated traffic. Precise and fair.
- IP address — necessary for unauthenticated endpoints like login, but blunt: an office or university behind one NAT shares an address, so a strict IP limit locks out an entire building.
- Endpoint — different costs deserve different limits.
POST /searchthat hits the database should not share a budget withGET /health.
Layer them. A global per-IP limit stops crude floods; a per-user limit enforces your plan tiers; a tight per-endpoint limit protects the two or three routes that are genuinely expensive.
The mess: limits that do not work distributed
You implement a clean token bucket in memory. It works perfectly in testing. You deploy to ten servers behind a load balancer and your 100/minute limit quietly becomes 1,000/minute, because each server is counting its own tenth of the traffic and none of them know about the others.
This is the single most common rate limiter mistake, and it only appears in production.
The fix: shared state, with an escape hatch
Move the counter to a store every instance can see — Redis is the standard choice, using atomic increments so ten concurrent servers cannot lose an update to a race.
Two refinements worth naming in an interview:
- Run the check as a single atomic operation (a Lua script or an atomic increment-with-expiry), never read-then-write. Read-modify-write across the network is a race by construction.
- Decide what happens when Redis is unavailable. Fail open (allow traffic) and an outage of your limiter becomes an outage of your database. Fail closed (reject everything) and a Redis blip takes down your entire API. Most teams fail open with a conservative local fallback limit — and the fact that you have thought about it at all is the answer being scored.
Tell the client what happened
Return 429 Too Many Requests with a Retry-After header, plus X-RateLimit-Limit, X-RateLimit-Remaining and X-RateLimit-Reset. A well-behaved client can then back off correctly instead of hammering you harder. A limiter that rejects without telling clients when to return causes retry storms — it converts one problem into a worse one.
Quick check
You set 100 requests per minute per user. Users complain that opening the dashboard fails, even though it makes only 30 requests. What went wrong?
(Think about it before reading on.)
You used an algorithm with no burst allowance, or the limit is being applied per-connection rather than per-user. Thirty requests firing in 200 ms is normal page-load behaviour. A leaky bucket draining at ~1.6 requests per second will reject most of them. A token bucket sized at 100 lets the burst through and still holds the long-run average at 100/minute — same limit on paper, completely different user experience.
How to talk about rate limiting in an interview
"I'd put the limiter at the API gateway, before anything expensive. Token bucket per API key, because real clients burst on page load and I want to allow that while capping the average. Counters live in Redis behind an atomic script so the limit holds across all instances rather than being multiplied by the instance count. Unauthenticated endpoints like login also get a per-IP limit, kept loose because of NAT. Rejections return 429 with Retry-After so clients back off instead of retrying harder. If Redis is unreachable I fail open with a conservative local limit — I'd rather serve traffic than take the API down to protect it."
That covers placement, algorithm choice with a reason, the distributed problem, the client contract, and the failure mode. It is the complete answer in under a minute.
Key Takeaways
- Rate limiting protects shared capacity from any single client — usually a bug, not an attack.
- Enforce at the edge, before the request reaches your application or database.
- Token bucket is the usual default: it permits real bursts while capping the long-run average.
- Fixed windows are cheap but allow a double burst at the boundary.
- In-memory counters multiply by your instance count — use shared, atomic state.
- Always return 429 with Retry-After, and decide in advance whether you fail open or closed.
Next Steps
Rate limiting keeps traffic off your database. Database indexing is how you make the traffic that does get through cheap — and it is the difference between a query that scans ten million rows and one that touches ten.