System Design Primer · llms.txt
Beyond the primer: design questions
Written for this site, not part of The System Design Primer. The sources it draws on are listed under Further reading.
Almost every product sends notifications: a one-time login code, "your order shipped", "your friend commented", "presale starts in an hour". Each one is a small message, but the system behind them has to talk to several outside providers, respect what each user agreed to receive, survive provider outages and absorb sudden campaign spikes, all without delaying the login code behind a million marketing emails.
Interviewers like this question because it's mostly about asynchronous processing done carefully. Expect to cover queues, retries, deduplication, rate limiting, scheduling and the messy details of real channels. There's no clever algorithm at the centre, so the marks go to clear separation of concerns and good answers to "what happens when this fails?"
Pick a concrete product so the numbers mean something. Here, it's an event-ticketing app: people buy concert and sports tickets, follow artists and teams, and get reminders before events.
Functional
Non-functional
Estimates (assumed, round numbers)
Producers call one internal API. They never talk to APNs or an email provider directly.
POST /v1/notifications
{
"idempotency_key": "ticket-transfer-55120-accepted",
"category": "transactional", // security | transactional | reminder | social | marketing
"template": "ticket_transfer_accepted",
"recipients": { "user_ids": ["u_19"] },
"data": { "event_name": "...", "seat": "..." },
"priority": "high", // high | normal | bulk
"send_at": null, // or an absolute time
"expires_at": "2026-05-02T18:00:00Z" // don't deliver after this
}
-> 202 Accepted { "notification_id": "n_7f3a..." }
POST /v1/campaigns { audience, template, send_at_local: "10:00", ... }
GET /v1/notifications/{id} per-channel status
POST /v1/users/{id}/devices { platform: "ios", token: "..." }
PUT /v1/users/{id}/preferences
GET /v1/users/{id}/inbox?cursor=...The API returns 202 because sending is asynchronous. The idempotency_key lets a producer retry safely after a timeout, since the service stores keys for a day and returns the original notification_id on a repeat. expires_at stops a stale message from going out late. A login code delivered after 20 minutes is worse than none.
Main tables:
| Table | Key | Notable fields |
|---|---|---|
contacts | user_id | email, phone, verified flags, IANA time zone, locale |
devices | device_id (indexed by user_id) | platform, push token, app version, last_seen, active |
preferences | (user_id, category, channel) | enabled, consent source and timestamp |
templates | (template_id, version, channel, locale) | subject, body, status (draft, approved) |
notifications | notification_id | idempotency key, category, template version, data |
deliveries | (notification_id, user_id, channel) | status, provider message id, attempts, timestamps |
inbox | (user_id, created_at) | title, body, link, read_at |
Preferences are stored as overrides on top of per-category defaults, so a new user needs no rows at all. Keep the consent source and time for marketing channels. Laws on marketing consent differ by country, and when a regulator asks, you need to show why a user got a message.
notifications row and returns 202. Future sends go to the scheduler. Audience sends become a fan-out job that pages through the audience in batches (say 1,000 users at a time) and emits one task per user.suppressed, with the reason, so "why didn't I get it?" has an answer.deliveries and feeds analytics.Why a separate queue per channel? Isolation. If the SMS provider slows down or goes out, SMS messages back up in their own queue while push and email carry on. Each pool also scales and retries according to its provider's rules.
apns-collapse-id, FCM's collapse_key) for messages where only the latest matters, such as "gate changed". When a provider reports a token as unregistered or invalid, deactivate it right away. Stale tokens waste capacity on sends that can never arrive.List-Unsubscribe header with one-click unsubscribe (RFC 8058), which Gmail and Yahoo now require from bulk senders.A presale campaign puts 16 million messages on the push and email queues within minutes. If a login code joins the back of that queue, the user waits 15 minutes and gives up. So split by priority as well as channel:
Give each priority its own queue, or topic, per channel, with dedicated or weighted consumers, so bulk can never starve high. How you build this depends on the broker. With Kafka you'd use separate topics, since a partition is strictly ordered and has no notion of priority. RabbitMQ supports priority queues natively, but separate queues are still easier to reason about and to scale independently. Bulk consumers also run behind a token bucket set below the provider's quota, so a campaign never uses up the quota that high-priority traffic needs.
Classify every provider response:
sent, with the provider's message id.Retry-After when the provider sends it. Cap the attempts, and stop retrying once expires_at has passed.failed and clean up the token or address.Messages that use up their retries go to a dead-letter queue for inspection, not a loop. Reliability patterns covers backoff, jitter and circuit breakers. A circuit breaker per provider stops thousands of workers hammering a provider that's already down.
Queues deliver at least once, so the same message can reach a worker twice. A worker can also crash after the provider accepted a message but before it recorded the fact. Defences, in layers:
notification_id + user_id + channel. Before calling the provider, the worker does a conditional write that moves the delivery row from queued to sending, and only one worker can win. A crashed sending row is retried only after a timeout.There's still a small window between "provider accepted" and "we recorded it" where a crash causes a duplicate. True exactly-once delivery to a phone isn't possible across systems you don't control, so say so in the interview and keep the window small.
These limits protect users, not just servers:
A sliding-window counter per user and category in Redis is enough. See rate limiting for the algorithms.
"Send at 10:00 local time" means a different UTC instant for each user. Store each user's IANA time zone (such as Asia/Kolkata, never a fixed offset, because daylight saving time changes offsets). Then compute each user's send time in UTC when you expand the campaign. Local times span from UTC−12 to UTC+14, so a "10:00 local" campaign naturally spreads over about 26 hours, which conveniently flattens the load.
Store scheduled sends bucketed by minute. A scheduler claims due rows and hands them to the router. In PostgreSQL, SELECT … FOR UPDATE SKIP LOCKED lets several scheduler instances claim rows without blocking each other. A delay queue is another option, but check the limits: Amazon SQS, for example, can delay a message by at most 15 minutes, so it can't hold "next Tuesday" on its own.
Watch for daylight-saving edge cases. 02:30 local may not exist on the spring-forward day, and in autumn an hour repeats (02:30 happens twice in central Europe, 01:30 in the US). Use a time zone library's rules for both cases and test them.
Relative schedules ("2 hours before the event") must follow the event. If the concert moves, the reminder must move too, so key scheduled reminders by event id, not just by time.
Each status change is an event appended to a log (Kafka works well here), which feeds two consumers: the deliveries row for per-message status, and a columnar analytics store for campaign metrics.
How much you can trust each step varies by channel. "Delivered" is solid for in-app, available from some SMS carriers, and mostly absent for push. Your own app can report "received" and "opened" back with the notification id. Email opens are counted with a tracking pixel, and they're unreliable. Some mail clients, notably Apple Mail with Mail Privacy Protection, load images automatically, which inflates open counts. Clicks, tracked through redirect links, are a more honest engagement signal. Feed engagement back into sending: stop marketing email to addresses that haven't opened or clicked in months, which also protects your sender reputation.
deliveries and inbox by user_id so reads for one user stay on one shard, and expire rows after 90 days. Write analytics events to a time-partitioned store, kept separate from operational tables.Login codes are probably sharing a queue, workers or provider quota with the campaign, so they wait behind bulk traffic. Give high-priority traffic its own queue per channel with dedicated workers, and put bulk consumers behind a token bucket below the provider quota, leaving room for high-priority sends. Also alert on the age of the oldest message in the high-priority lane, so you catch this before users do.
The queue redelivers the message, and another worker may send it again, causing a duplicate. The conditional queued → sending write stops two workers sending in parallel. But a row stuck in sending after a crash has to be retried eventually, so a small duplicate window remains. Keep it small by recording right after the provider call and setting a sensible timeout before retrying sending rows. For truly sensitive messages, accept the rare duplicate rather than risk losing the message.
India (UTC+5:30) sends at 03:30 UTC, Germany at 07:00 or 08:00 UTC depending on daylight saving, and California at 16:00 or 17:00 UTC. So the load arrives in three separate waves, not one spike. If you store fixed offsets instead of IANA zones, users in Germany or California will get it an hour off after the clocks change. Always compute from the zone name at expansion time.
At 600 / 100 = 6 workers at full use, so run 8–10 to leave headroom for retries and slow responses. The real ceiling is usually the provider's account-level throughput, plus per-country or per-sender-number limits, which no number of workers gets around. Spread traffic across approved sender numbers or short codes where the rules allow, and make sure per-user limits stop runaway retries from burning money.
The message was probably already past the router, either sitting in the push queue or rendered by a worker, before the preference change landed. Or the router read a cached preference that hadn't expired yet. Re-checking preferences in the worker just before sending closes most of the gap, at the cost of one more lookup per message. A short window is usually acceptable for marketing, but invalidate caches on change and keep the window to seconds.
apns-collapse-id and apns-expiration, plus response codes.FOR UPDATE SKIP LOCKED for job claiming.Previous: Design a distributed key-value store · Next: Design a chat system