A rate limiter caps how many requests a client can make in a period of time and turns away the rest. It protects shared capacity from one noisy client, blunts abuse like credential stuffing and scraping, keeps costs predictable when each request costs you money (SMS, third-party APIs, GPU time), and turns API pricing tiers into something you can enforce.
\nInterviews use it two ways. \"Design a rate limiter\" is a classic standalone question, and it tests whether you can pick an algorithm, keep counters correct across many servers, and decide what happens when the limiter itself fails. Rate limiting also comes up as a component inside bigger designs: any public API, login flow or notification system should have one.
\nIt's related to back pressure in the primer, but the focus differs. Back pressure reacts to how loaded the server is. A rate limiter enforces a policy per client, whether or not the server is busy.
","headings":[]},{"id":"where-the-limiter-lives","title":"Where the limiter lives","html":"You can enforce limits at several layers, and each one sees different information.
\nMost production systems layer these: a coarse per-IP limit at the edge to absorb floods, per-key quotas at the gateway, and a few cost-based limits inside expensive services.
\nThe limiter counts requests per key, and the key you choose decides who gets hurt by a false positive.
\nReal rules combine these, and a request must pass all of them. A login endpoint, for example, might allow 5 failed attempts per username per 15 minutes (to stop password guessing against one account) and 100 per IP per hour (to slow credential stuffing across many accounts).
","headings":[{"id":"choosing-the-key","title":"Choosing the key","depth":3}]},{"id":"the-algorithms","title":"The algorithms","html":"The five common algorithms are easiest to understand as a progression. Each one fixes a weakness of the one before it, and the last two take a different view: instead of counting requests in a window, they model a rate directly.
\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n| Algorithm | State per key | Bursts | Accuracy |
|---|---|---|---|
| Fixed window counter | one counter | up to 2× the limit across a boundary | coarse |
| Sliding window log | one timestamp per request | none past the limit | exact |
| Sliding window counter | two counters | small overshoot possible | approximate |
| Token bucket | token count + last refill time | allows bursts up to bucket size | exact for its rule |
| Leaky bucket (queue) | a bounded queue | smoothed to a fixed output rate | exact; adds delay |
Keep one counter per key per window (per minute, say), increment it on each request and reject once it passes the limit. It's the cheapest option and easy to reason about. The flaw is at the boundary. With a limit of 100 a minute, a client can send 100 requests at 12:00:59 and 100 more at 12:01:00, which is 200 in about a second. Clients that were rejected also tend to retry together when the window resets, causing a spike.
\nStore a timestamp for every accepted request. On each new request, drop timestamps older than the window, count what's left and allow the request if the count is under the limit. It's exact, but memory grows with the limit: a limit of 10,000 an hour means storing up to 10,000 timestamps per key. It suits small limits where precision matters.
\nKeep fixed-window counters for the current and previous windows, and weight the previous one by how much of it still overlaps the sliding window:
\nestimate = current + previous × (1 − elapsed fraction of current window)Say the limit is 50 a minute. The previous minute saw 48 requests, and 40 seconds into this minute there have been 31. Two-thirds of the window has elapsed, so a third of the previous minute still counts: 31 + 48 × ⅓ = 47. The request is allowed, and so are the next two; the one after that is rejected. This assumes the previous window's requests were spread evenly, so it can be slightly off. It costs two counters per key, and it removes most of the boundary burst. Cloudflare described using this approach and found the error small in practice.
\nEach key has a bucket that holds up to capacity tokens and refills at rate tokens per second. A request takes one token, or more if it's expensive. No token, no service. A client that's been quiet can burst up to the full capacity, while the long-run average can't exceed the refill rate. Two knobs, burst and sustained rate, map well onto how people actually use APIs, which is why it's the most common choice for public APIs (Stripe describes its token-bucket limiters in the post under Further reading).
You don't need a timer to refill. Store the time of the last update and add the tokens owed whenever a request arrives:
\nimport time\n\nclass TokenBucket:\n def __init__(self, rate: float, capacity: float):\n self.rate = rate # tokens added per second\n self.capacity = capacity # largest burst allowed\n self.tokens = capacity # start full\n self.updated = time.monotonic()\n\n def try_acquire(self, cost: float = 1.0) -> float:\n \"\"\"Return 0 if allowed, otherwise the seconds to wait before retrying.\"\"\"\n now = time.monotonic()\n # Refill lazily from the elapsed time.\n self.tokens = min(self.capacity, self.tokens + (now - self.updated) * self.rate)\n self.updated = now\n if self.tokens >= cost:\n self.tokens -= cost\n return 0.0\n return (cost - self.tokens) / self.rateThe wait time it returns is exactly what you'd put in a Retry-After header. This version is single-process and not thread-safe; the distributed version comes later.
There are two versions of the name. As a meter, it's the token bucket viewed from the other side: a bucket that fills with each request and drains at a fixed rate, rejecting when full. As a queue, requests wait in a fixed-size FIFO and are released at a constant rate, and new requests are dropped when the queue is full. The queue form smooths traffic into a steady stream, which protects a backend that can't handle bursts, at the price of added latency. NGINX's limit_req uses this method. Its burst setting sizes the queue, and nodelay passes queued requests through immediately, which makes it behave much like a token bucket.
With one gateway instance, the counters can live in memory. With twenty, they can't: each instance would see a twentieth of a client's traffic and enforce a limit twenty times too loose. The usual fix is a shared, fast store, most often Redis.
\nThe race. The naive version reads the count, checks it and writes it back. Two gateways can both read 99, both decide the request is under the limit of 100, and both write 100. Both requests get through, and the counter is now wrong. Under real concurrency this happens constantly.
\nAtomic increments fix the simple case. INCR increments and returns the new value in one step, so check the result after incrementing: if INCR returns 101, reject. For a fixed window, put the window number in the key name (for example rl:user42:28732150) and send INCR and EXPIRE together in a MULTI/EXEC transaction so the key can't be left without an expiry if the client crashes between the two commands. Because the window is part of the name, refreshing the expiry on every request is harmless.
Lua scripts fix the rest. A token bucket or sliding log needs to read state, decide and write several fields. Redis runs a Lua script atomically: no other command runs while it executes. Here is a sliding-window log as one script:
\n-- KEYS[1]: the limit key. ARGV: now_ms, window_ms, limit, unique request id\nlocal now = tonumber(ARGV[1])\nlocal window = tonumber(ARGV[2])\nlocal limit = tonumber(ARGV[3])\n\nredis.call('ZREMRANGEBYSCORE', KEYS[1], '-inf', now - window)\nif redis.call('ZCARD', KEYS[1]) < limit then\n redis.call('ZADD', KEYS[1], now, ARGV[4])\n redis.call('PEXPIRE', KEYS[1], window)\n return 1\nend\nreturn 0The request ID must be unique, because a sorted set holds each member once, and two requests in the same millisecond with the same member would count as one. The time comes from the caller here, so gateway clocks need to agree. Alternatively, read Redis's own clock with TIME inside the script, which Redis 5 and later allow.
Other things to plan for:
\n{user42} if you need several.Status code. Use 429 Too Many Requests, defined in RFC 6585, when a client exceeds its limit. Some services use 503 Service Unavailable when the whole system is overloaded rather than one client. The distinction helps clients: a 429 means \"you slow down\", a 503 means \"everyone wait\".
When to retry. Retry-After (defined in HTTP semantics, RFC 9110) gives either a number of seconds or a date. Seconds are safer, because they don't depend on the client's clock being right.
Remaining quota. There's no finished standard yet. Many APIs use unofficial X-RateLimit-* headers. GitHub, for example, sends x-ratelimit-limit, x-ratelimit-remaining, x-ratelimit-used and x-ratelimit-reset (a Unix timestamp). The IETF HTTP APIs working group is standardizing this in draft-ietf-httpapi-ratelimit-headers. Early versions of the draft defined separate RateLimit-Limit, RateLimit-Remaining and RateLimit-Reset fields. Recent versions use two structured fields: RateLimit-Policy describes the quota, and RateLimit reports what's left. Since it's still a draft, check the current version before copying the syntax.
HTTP/1.1 429 Too Many Requests\nRetry-After: 7\nRateLimit-Policy: \"per-key\";q=100;w=60\nRateLimit: \"per-key\";r=0;t=7\nContent-Type: application/json\n\n{\"error\": \"rate_limited\", \"message\": \"Limit of 100 requests per minute exceeded.\"}A few habits make limits easier to live with:
\nUp to 200: 100 at the very end of one window and 100 at the very start of the next. A sliding window log prevents it exactly. A sliding window counter prevents most of it, at the cost of two counters. A token bucket with a capacity of 100 still allows a 100-request burst, but not 200, because the bucket can't refill instantly.
\nThe bucket is full after the idle period, so 50 succeed and 30 are rejected. The bucket is then empty and refills one token every 0.1 seconds. Retry-After takes whole seconds, so you'd round up and send 1.
Both instances can read the same value, both decide the request is allowed, and both write the same incremented value. Requests slip past the limit and the count is too low. Use INCR and check the value it returns, or move the whole read-decide-write into a Lua script so Redis runs it atomically.
The read API should usually fail open, ideally falling back to a looser per-instance limit, since rejecting every request turns a limiter outage into a full outage. The SMS endpoint should fail closed: each request costs money, and attackers abuse unprotected SMS endpoints for toll fraud. The answer depends on whether the limit protects capacity or protects money and security.
\nThey're probably being limited per IP or per API key, and their backend sends all their end users' traffic through one IP and one key. Every user shares a single quota. You could raise that customer's tier, offer per-end-user limits (letting them pass a user identifier you trust), or document the limit so they can queue on their side.
\nRetry-After and 503.INCR.burst and nodelay.