System Design Primer · llms.txt
Written for this site, not part of The System Design Primer. The sources it draws on are listed under Further reading.
The primer's Asynchronism chapter explains why queues exist: put slow work on a queue and let workers pick it up later. Interviewers then ask the next questions. What happens if a worker crashes halfway through a message? Can two events for the same order be processed out of order? Can you replay last Tuesday's events after fixing a bug?
The answers depend on which of two designs you pick: a queue, where a message is handed to one worker and then deleted, or a log, where messages are appended to a durable sequence that many readers walk through at their own pace. This chapter covers both, the delivery guarantees they can give, and the operational pieces (dead-letter queues, back pressure, partition keys) that come up in almost any design with an asynchronous step.
RabbitMQ and Amazon SQS are the usual examples. The broker hands each message to one consumer and deletes it when the consumer acknowledges it. If the consumer doesn't acknowledge in time, the message becomes available again and another consumer can take it. SQS calls this window the visibility timeout; RabbitMQ redelivers unacknowledged messages when the consumer's channel or connection closes.
This model is good at spreading independent jobs across a pool of workers: add workers and throughput goes up, because any worker can take any message. The cost is that an acknowledged message is gone. There is nothing to replay, and a second application that wants the same messages needs its own copy (a fanout exchange in RabbitMQ, or an SNS topic feeding several SQS queues).
Ordering is weak. With several workers on one queue, message 2 can finish before message 1. SQS standard queues promise only best-effort ordering. SQS FIFO queues keep order within a message group ID, and RabbitMQ delivers in order from a single queue, but parallel consumers and redeliveries can still reorder processing.
Apache Kafka and Amazon Kinesis Data Streams work differently. A topic (Kinesis calls it a stream) is split into partitions (Kinesis: shards). Each partition is an append-only log. Every record gets a sequential offset within its partition, and records are not deleted when someone reads them. They stay until a retention limit expires: a time limit, a size limit, or, in Kafka, log compaction, which keeps only the latest record for each key.
Readers form consumer groups. Within a group, each partition is assigned to exactly one consumer, so the group splits the work. Each group tracks its own position (the offset it has processed up to), so two groups reading the same topic don't interfere.
Four consequences follow:
Suppose a delivery app streams courier GPS pings. Assume 200,000 active couriers, each sending one ping every 4 seconds, at about 200 bytes per ping.
The shape of the reasoning is what matters: throughput gives bandwidth, retention gives storage, and consumer speed gives the minimum partition count.
The question is what happens when a consumer fails between "I received this" and "I finished this".
| Guarantee | How the consumer behaves | What can go wrong |
|---|---|---|
| At-most-once | Acknowledge (or commit the offset) before processing | A crash after the ack loses the message |
| At-least-once | Process, then acknowledge | A crash after processing but before the ack causes a redelivery, so the work happens twice |
| Effectively-once | At-least-once delivery plus processing that tolerates duplicates | Correctness now depends on your dedup logic |
Assume at-least-once for SQS, RabbitMQ with manual acks, and Kafka consumers that commit after processing. Duplicates are routine: they follow consumer crashes, rebalances, and visibility timeouts that expire because a job ran long.
Over an unreliable network, a sender can't know for sure whether a message arrived if the acknowledgment is lost, so it must either resend (risking a duplicate) or not (risking a loss). Exactly-once delivery in that sense isn't achievable. What systems offer instead is exactly-once processing: the effect of each message is applied once, even though the message itself may be delivered more than once.
There are two common ways to get there:
read_committed. Kafka's idempotent producer also stops a producer retry from writing the same record twice.Kafka's exactly-once guarantee stops at Kafka's edge. If your consumer sends an email or calls a payment API, you are back to idempotency on the receiving side. Likewise, SQS FIFO queues deduplicate sends within a 5-minute window, which protects against producer retries but not against your consumer's side effects running twice.
In an interview, "we use at-least-once delivery with idempotent consumers" is almost always the right answer. Say how the consumer detects duplicates.
Some messages will never succeed: a malformed payload, a reference to a deleted record, a bug that only triggers on certain input. With at-least-once delivery, such a poison message is redelivered forever, burning worker time, and in a log it blocks every record behind it in the same partition.
A dead-letter queue (DLQ) is where these messages go after a set number of failed attempts. SQS does this with a redrive policy and maxReceiveCount. RabbitMQ uses a dead-letter exchange, and quorum queues can enforce a delivery limit. Kafka has no built-in DLQ for plain consumers; you write failed records to a separate topic yourself (Kafka Connect and many client frameworks provide this as an option).
A DLQ only helps if someone looks at it. Alert on its depth, and have a way to redrive messages back to the main queue once the bug is fixed. Also check what skipping does to ordering: if events for one order must be applied in sequence, parking event 3 and applying event 4 may leave the order in a wrong state.
The primer's back pressure section covers bounding queue size and returning 503s. Two points to add.
Push and pull brokers behave differently. RabbitMQ pushes messages to consumers, so you limit flow with a prefetch count, the number of unacknowledged messages a consumer may hold. Kafka and Kinesis consumers pull, so a slow consumer just falls behind.
Falling behind in a log has a hard limit. The gap between the newest offset and a group's committed offset is consumer lag, and it is the main health metric for a streaming system. If lag grows past the retention period, the oldest unread records are deleted before the consumer reaches them, and that data is lost. Monitor lag in time ("we are 40 minutes behind") as well as in records, and alert well before it approaches retention.
A buffer absorbs bursts, not a producer that is permanently faster than its consumers. That needs more consumers (and partitions), less work per message, or slower producers.
In a log, the partition key decides ordering, parallelism and load balance at once. Kafka's default partitioner hashes the key and takes it modulo the partition count, so all records with the same key land in the same partition, in order.
order_id, so "created", "paid", "shipped" for one order arrive in sequence. Events for different orders can interleave freely.tenant_id + user_id), or split the hot key with a suffix and accept losing strict ordering for it.country with 30 values can't spread evenly across 48 partitions.hash(key) mod N, changing N moves most keys. Old records stay put, so for a while one key's events sit in two partitions and per-key ordering can break. Kafka's default partitioner doesn't use consistent hashing.| You need | Lean towards |
|---|---|
| A pool of workers doing independent jobs (resize image, send email) | A queue: SQS, RabbitMQ |
| Per-message retry, delay, priority, or routing rules | A queue: RabbitMQ's exchanges and SQS's delay and visibility settings fit this |
| Several independent consumers of the same events | A log: Kafka, Kinesis, Pulsar |
| Replay after a bug, or backfilling a new service from history | A log, with retention sized for the replay window |
| Strict ordering per entity at high throughput | A log, keyed by entity |
| High volume (tens of thousands of events per second and up) with steady throughput | A log |
| Low operational effort at small scale | A managed queue such as SQS |
The lines are blurring. RabbitMQ added streams, a log-style data structure with offsets and replay. Redis Streams has consumer groups. Pulsar supports both queue-style and stream-style subscriptions. Kafka itself is adding share groups (KIP-932, "Queues for Kafka"), which let several consumers share a partition and acknowledge records one at a time, as a queue would. In an interview, name the model you want (queue or log) and the property that made you choose it, then name a product.
Two mistakes are common. Using Kafka for a simple background job queue brings partition counts, rebalances and head-of-line blocking that a work queue wouldn't have. Building an event-driven system where several services react to the same events on a queue that deletes messages once read means you can't replay or add a consumer later without copying data around.
Only 12 of the 20 instances are working, because each partition goes to one consumer in the group, so adding instances won't help. Increase the partition count (accepting that keys get remapped), make each consumer faster (for example by batching database writes), or process records in parallel inside a consumer while committing offsets in order.
After restart, the group resumes from the last committed offset, so the event is processed again and the merchant is credited twice. To make it safe, record the event ID (or the partition and offset) in a table in the same database transaction as the credit, and skip events whose ID is already there. Kafka transactions don't help here, because the side effect is in Postgres, not Kafka.
The three partitions holding those cities will be hot, and their consumers will lag while others sit idle. Ask what actually needs ordering. Usually it's the events for one ride, so key by ride_id. That spreads load evenly and still keeps each ride's events in order.
SQS deletes messages once processed, and keeps even unread ones for at most 14 days, so last month's events are gone. You'd need the raw events archived elsewhere, in object storage or in a log like Kafka or Kinesis with long enough retention. It's a good argument for putting events several teams care about on a log.
The consumer can't move past that offset, so every record behind it in the partition waits. That is head-of-line blocking. Catch the failure after a few retries, write the record and its error to a dead-letter topic, commit past it, and alert so someone can fix and replay it. Check first whether skipping it breaks ordering assumptions for that key.
maxReceiveCount in practice.Previous: Unique ID generation · Next: Timeouts, retries and idempotency