The signup that takes eleven seconds
A user clicks "Create account". Your handler creates the row, sends a welcome email, generates an avatar thumbnail, syncs to the CRM, and fires an analytics event. Then it returns 200.
The database write took 20 ms. The other four steps took eleven seconds. Your user stared at a spinner for eleven seconds to complete an operation that was actually finished in twenty milliseconds.
Worse: when the CRM has an outage, signup fails. A marketing tool that has nothing to do with authentication can now stop people from joining your product. That is the real bug — the coupling, not the latency.
The analogy: the restaurant order rail
Watch a busy kitchen. The waiter does not stand at the pass waiting for the food. They write the order, clip it to a rail, and immediately go serve the next table. Cooks pull tickets off the rail at their own pace.
The rail does three things at once. It decouples the waiter from the cook — neither waits for the other. It buffers the dinner rush, holding tickets when orders arrive faster than the kitchen can cook. And it preserves the work — if a cook drops a pan, the ticket is still clipped to the rail.
A message queue is that rail. The waiter is your API. The cooks are your background workers.
What actually changes
Rewrite the signup with a queue and the handler becomes: create the user row, publish a user.created event, return 200. Response time goes from eleven seconds to about 25 ms.
Four separate consumers subscribe to that event and do their jobs independently. When the CRM is down, its consumer retries — the other three are unaffected, and signup still works. You have converted a chain that fails as a unit into four things that fail alone.
Queues vs. topics: the distinction interviewers listen for
A queue (point-to-point) delivers each message to exactly one consumer. Ten workers reading one queue split the work between them. Use it for tasks: resize this image, charge this card.
A topic (publish/subscribe) delivers each message to every subscriber. One user.created event reaches the email service, the analytics service, and the CRM service — each getting its own copy. Use it for facts: this thing happened.
Getting this backwards is a common interview stumble. "Send a welcome email" is a task for a queue. "A user was created" is a fact for a topic. Model events as facts and adding a fifth consumer later requires no change to the publisher at all.
Want to master this with video lessons and real case studies? This topic is covered in depth in my Udemy course System Design Fundamentals for Interviews — 5.5 hours, rated 4.8★, built from real interview questions.
Delivery guarantees, and the one you actually get
There are three, and only two exist in practice:
- At-most-once — fire and forget. Fast, and you will lose messages. Fine for non-critical metrics, wrong for payments.
- At-least-once — the broker redelivers until the consumer acknowledges. Nothing is lost, but duplicates are guaranteed. This is what almost every real system uses.
- Exactly-once — what everyone wants. Across a network with retries it is not achievable in the general case; what brokers offer is at-least-once delivery plus deduplication, which is a different thing wearing the same label.
So the honest position is: you get at-least-once, and you make duplicates harmless.
The mess: the double charge
A consumer picks up charge_card(order_123, $50), calls the payment provider successfully, and then the worker crashes before it can acknowledge the message.
The broker sees no acknowledgement. It assumes the work was never done, so it redelivers. A second worker charges the card again. Your customer has paid $100 for a $50 order, your code has no bug in it anywhere, and every component behaved exactly as designed.
The fix: idempotent consumers
Make processing the same message twice have the same effect as processing it once.
Give every message a stable ID — an order ID, or a UUID the producer generates. Before doing the work, the consumer records that ID; if the ID is already recorded, it acknowledges and does nothing. The cleanest version writes the ID and the result in the same database transaction as the work itself, so there is no window where one happened without the other.
The rule to carry into every interview: with at-least-once delivery, idempotency is not optional. If a candidate proposes a queue and does not mention it, that is the first thing a good interviewer will probe.
Dead letter queues
What about a message that fails every single time — malformed payload, a deleted record, a bug in the consumer? Retried forever, it blocks the queue and burns capacity permanently.
A dead letter queue is the escape hatch. After N failed attempts, the broker moves the message to a separate queue where it stops being retried and starts being visible. Your main pipeline keeps flowing, and you get an alert plus a durable copy of the exact payload that broke.
An unmonitored DLQ is a silent data-loss bucket, so alert on its depth. A DLQ that has been quietly filling for three weeks is worse than no DLQ at all.
Quick check
Your queue depth is growing steadily and never drains. Producers publish 1,000 messages a second; consumers process 600. What do you do?
(Think about it before reading on.)
Nothing about the queue will fix this. A queue absorbs bursts, not a permanent deficit — if the average arrival rate exceeds the average processing rate, backlog grows without limit and the buffer only decides how long before you notice. Either add consumers until throughput exceeds 1,000/s, or make each message cheaper to process. This is the same principle as the narrowest stage setting your throughput: a buffer in front of a slow stage does not make the stage faster.
How to talk about queues in an interview
"Signup only needs the user row to be durable before we respond. Everything else — email, thumbnails, CRM sync — is a side effect, so I'd publish a user.created event to a topic and let independent consumers handle it. That takes response time from seconds to milliseconds and stops a CRM outage from breaking signup. Delivery is at-least-once, so consumers are idempotent on the user ID, and anything that fails repeatedly lands in a dead letter queue with an alert on depth. If the backlog grows persistently rather than in bursts, that's a capacity problem, not a queue problem — I'd scale consumers."
Queues also show up constantly in case studies. In a coding contest platform, the queue between submission and judging is the whole design — it is what lets the API answer in milliseconds while 50,000 submissions arrive in ten seconds.
Key Takeaways
- Queues decouple producers from consumers, buffer spikes, and preserve work across crashes.
- Queues deliver to one consumer; topics deliver to every subscriber. Tasks go on queues, facts go on topics.
- You get at-least-once delivery in practice, which means duplicates are certain.
- Idempotent consumers keyed on a stable message ID are what make duplicates harmless.
- Dead letter queues stop poison messages from blocking the pipeline — alert on their depth.
- A queue absorbs bursts, not a permanent throughput deficit.
Next Steps
Queues protect you from your own workload. But what protects you from someone else's — a scraper, a runaway retry loop, a botnet? That is rate limiting, and the algorithm you pick decides whether legitimate bursts survive.