System Design Primer · llms.txt
Written for this site, not part of The System Design Primer. The sources it draws on are listed under Further reading.
The primer starts beginners off with two classics: David Malan's Scalability lecture from Harvard's CS75 (summer 2012), and Sebastian Kreutzberger's four-part blog series Scalability for Dummies (2011). Both still hold up. The lecture runs close to two hours, though, and the blog names tools and habits from more than a decade ago. This chapter covers the same ground in one sitting: vertical and horizontal scaling, clones, load balancing, caching, replication, partitioning and asynchronism. Along the way it shows real companies that hit each wall, with sources you can check. Read it instead of the originals, before them, or after them as a refresher.
To keep things concrete, we'll follow one imaginary app as it grows. Call it Crumb, a recipe-sharing site. It begins as a weekend project. Then a food influencer with four million followers posts a Crumb link, and the traffic arrives overnight.
Before that, one definition. A system is scalable if adding resources gets you a matching gain in capacity. If one server handles 1,000 requests a second and ten servers handle about 10,000, the system scales. If ten servers still handle only 1,500 because they all wait on one database, it doesn't. The primer's chapter on performance vs scalability puts it this way: a performance problem makes the system slow for one user, and a scalability problem makes it fast for one user but slow under load.
Crumb launches the way most products do. One rented server runs the web app, the database and the uploaded photos.
That's the right call at the start. Facebook began the same way: when it launched from a Harvard dorm room in February 2004, it ran on a single rented server that cost $85 a month. Building for millions of users before you have a hundred is wasted effort, and a simple system is easier to change once you learn what users actually do.
The trouble is that one box is a single point of failure, and it has a ceiling. When the influencer's post goes out, you have two ways to raise that ceiling.
Vertical scaling (scaling up) means moving to a more powerful machine: more CPU cores, more RAM, faster disks. Your code doesn't change, and nothing new can go wrong between machines. For many workloads this goes further than people expect.
Stack Overflow is the standard example. In February 2016, Nick Craver described their setup: about 209 million HTTP requests a day, served by 11 web servers and 4 SQL Server machines. The database servers held 384 GB and 768 GB of RAM each, enough to keep most of the data in memory. He also wrote that they were "down to needing only 1 web server", and that they had "unintentionally tested this, successfully, a few times". That's what tuning plus big hardware can do.
Vertical scaling has three limits, and the CS75 lecture covers all of them:
Crumb upgrades its server, which buys a few weeks. Then the next jump in traffic arrives.
Horizontal scaling (scaling out) means running many ordinary machines in place of one big one. Put a load balancer in front, and each request goes to whichever server is free. If you need more capacity, add servers. If one dies, the others carry on.
There's a catch, and it's the subject of the first part of Kreutzberger's series. Say user Steve logs in on server 2, and his next request lands on server 3. If server 2 kept his login session in its own memory, server 3 doesn't know him, and he's logged out for no reason. His uploaded photo saved to server 2's disk is a broken image on server 3.
Kreutzberger's fix, which he called the golden rule of scalability: every server runs exactly the same code and keeps no user data on its own disk or memory. Sessions go to a shared store such as Redis or the database. Uploads go to shared storage, which today usually means an object store like Amazon S3. The servers become identical, disposable clones, also called stateless servers.
The lecture also covers the other option, sticky sessions: the load balancer sends a user back to the same server every time, often using a cookie. It's quicker to set up, but load spreads unevenly, and when a server dies, every user stuck to it loses their session. Stateless servers are the more robust default.
The 2011 series made clones by baking a server image (an AMI, on AWS) and deploying code to every server with a tool called Capistrano. The idea hasn't changed, but the tools have. Today the image is usually a container, an autoscaling group or Kubernetes starts and stops copies based on load, and a CI pipeline handles deploys. If one of your servers can't be deleted and replaced without anyone noticing, it isn't a clone yet.
Crumb moves sessions into Redis and photos into object storage, then runs three app servers behind a load balancer. One server can now crash during dinner rush without users noticing.
The load balancer is what makes a pool of clones look like one website. The lecture walks through how it evolved, and each step is still in use somewhere.
DNS round robin. The simplest version needs no load balancer. Your domain name returns a different server's IP address on each lookup. The problem is caching: browsers, operating systems and ISPs hold on to DNS answers, so one busy server can keep getting traffic, and a dead server keeps getting traffic until the cached records expire.
A dedicated load balancer. A piece of software or hardware sits in front of your servers and decides where each request goes. It runs health checks so it stops sending traffic to servers that fail them. Common ways to choose a server are round robin, least connections and hashing on the client's address. A layer 4 balancer routes on IP addresses and ports. A layer 7 balancer reads the HTTP request, so it can send /api to one pool and /images to another. The load balancer chapter covers both.
The load balancer itself can't become the new single point of failure, so production setups run a pair, one active and one on standby, or use a managed service the cloud provider keeps redundant.
A real-world test. When Pokémon GO launched in Australia and New Zealand in July 2016, Niantic had planned for a certain level of traffic, with a worst case of about five times that. According to Google Cloud, traffic blew past expectations within 15 minutes of launch and went on to hit 50 times the plan, ten times the worst case. The game survived because it was built to scale out: Google added tens of thousands of CPU cores to the game's Kubernetes cluster, and the team swapped the network load balancer for Google's global HTTP(S) load balancer, which handled the traffic better. Adding capacity that fast is only possible when servers are clones.
With clones and a load balancer, the app servers can grow without limit. The database, still a single machine, is now where load piles up. The cheapest way to relieve it is to stop asking it the same questions.
A cache is a fast key-value store, usually held in memory (Memcached or Redis), placed between your app and the database. The most common pattern is cache-aside:
Reading from memory takes well under a millisecond. A database query can take tens or hundreds of milliseconds. For a read-heavy app like Crumb, where every popular recipe is viewed thousands of times for each edit, most reads can come from the cache.
Kreutzberger's third part compares two ways to decide what to cache:
He also gave a warning that still holds: keep the cache in a shared in-memory store, not in files on each app server, or your clones stop being identical.
How far can this go? Facebook adopted memcached in 2005, and by 2013 its cache tier was serving billions of requests per second. Most reads on the site never reached a database.
What can go wrong. A cache that the database depends on is also a risk. On 23 September 2010, Facebook was down for about two and a half hours, its worst outage in four years. An automated system checked cached configuration values, and when it found an invalid one it fetched a fresh copy from the database. Then someone saved a value to the database that the system itself treated as invalid. Every client saw the bad value and queried the database to fix it, and the database cluster got hundreds of thousands of queries a second. Each failed query made things worse, and the only way out was to take the whole database cluster offline, which took the site down with it. The lesson applies to any cache: plan for what happens when many requests miss at once, a problem known as a cache stampede or thundering herd.
The cache chapter covers the other update strategies (write-through, write-behind, refresh-ahead) and the trade-offs of each.
Caching handles repeated reads. Writes, and reads the cache misses, still go to one database, and if it fails, Crumb is down. Replication keeps copies of the database on other machines.
The most common setup is primary-replica (the primer calls it master-slave). All writes go to the primary, which streams its changes to one or more replicas. Reads can go to any replica.
That buys you two things:
Instagram used this setup early on. In late 2011, with 14 million users and three engineers, its published architecture listed 12 PostgreSQL database servers, each streaming to a replica in a different availability zone. A whole data center could fail without losing data.
The catch: replication lag. Replicas apply changes a moment after the primary. Usually that's milliseconds, but under heavy load it can be seconds. A Crumb user posts a comment, the page reloads, the read goes to a replica that hasn't received the comment yet, and the comment seems to be missing. They post it again, and now there are two. The usual fix is read-your-own-writes: for a short time after a user writes something, send that user's reads to the primary.
The lecture also covers primary-primary (master-master) replication, where both nodes accept writes. It helps with availability, but conflicting writes become your problem. The primer's replication section lists the drawbacks of each setup.
Replicas spread out reads, but every write still goes to a single primary, and eventually the data won't fit on one machine. The next step is partitioning: splitting the data so different machines own different parts of it.
hash(user_id) % N, so data spreads evenly.Early Facebook used a version of this that fit its product well. In a 2005 guest lecture at Harvard's CS50, Mark Zuckerberg described giving each school its own MySQL database. Most activity happened between students at the same school, so most queries touched only one small database, and adding a school meant adding a machine.
That example shows the main lesson of sharding: choose a key that keeps related data together. Get it wrong and you get two problems:
Moving to a sharded database later is also hard, so teams shard only when they have to.
Kreutzberger's second part offered two paths: keep MySQL and add replicas, sharding and tuning as it strains, or denormalize early and move to a NoSQL database like MongoDB. That was common advice in 2011, and it has aged unevenly. Relational databases kept improving, managed services made replicas and failover routine, and many large sites still run on sharded MySQL or PostgreSQL. The part that holds up: plan for how the data could be split, and don't build queries that need the whole dataset in one place. The database chapter covers federation, sharding, denormalization and SQL vs NoSQL in detail.
Some work is slow no matter how many servers you have: resizing a 12-megapixel photo, sending 50,000 notifications, building a weekly report. If that work happens while the user waits, the request ties up a server for seconds, and a burst of uploads can fill every worker the app has.
Kreutzberger's fourth part explains this with a bakery. If you ask for bread and they tell you to come back in two hours, that's bad service. Bakeries avoid it in two ways, and so do websites.
Bake at night: do the work in advance. The bakery bakes bread before opening, so there's no wait. A website can precompute work too: render popular pages to static HTML, build each user's feed ahead of time, recompute "trending recipes" every five minutes instead of on every page view. Put the results on a CDN and they serve millions of visitors without touching your servers.
Take the order: queue the work. A custom birthday cake can't be made in advance, so the bakery takes the order and tells you when to collect it. On a website, the app puts a job on a queue and replies at once ("Your photo is processing"). A pool of workers takes jobs off the queue and does them. The user sees a fast response, and the slow part happens in the background.
Instagram's 2011 setup used this pattern too: around 200 Python workers took tasks from a Gearman queue, including pushing each new photo into followers' feeds and sharing posts to Twitter and Facebook. Today the queue is more likely to be RabbitMQ, Amazon SQS, Redis or Kafka, but the pattern hasn't changed.
Queues also absorb spikes. If Crumb gets 10,000 uploads in a minute and its workers handle 2,000 a minute, the queue holds the backlog and the workers catch up. Nothing crashes. Users just wait a bit longer for thumbnails.
Queues have limits too. In November 2022, Ticketmaster ran Taylor Swift's Eras Tour presale through Verified Fan, an invite-and-queue system designed to control how many people reach checkout at once. About 1.5 million fans were invited. Bots and fans without codes still sent 3.5 billion requests, four times Ticketmaster's previous peak. Ticketmaster had to slow and delay sales, and it later cancelled the general sale. A queue buys time, but if work keeps arriving faster than workers finish it, the queue only grows. At that point you need to turn requests away early, which the primer covers under back pressure.
The asynchronism chapter covers message queues and task queues in more depth.
A year later, Crumb looks like this. Every part has a job, and none is a single point of failure.
Each change fixed a specific problem. When you're designing a system, start with the symptom, not the tool:
| Symptom | Likely fix | Covered in |
|---|---|---|
| One server is maxed out, and you're early | Scale up to a bigger machine | Scaling up |
| You need more capacity, or can't afford downtime | Stateless clones behind a load balancer | Scaling out |
| The database is busy answering the same reads | Cache in memory | Caching |
| Too many reads for one database, or it's a single point of failure | Read replicas and failover | Replication |
| Too many writes, or too much data for one machine | Federation or sharding | Partitioning |
| Requests are slow because of heavy work | Precompute it, or queue it for workers | Asynchronism |
Two warnings to finish. First, every fix adds parts, and parts fail. A single server is easy to reason about, and five layers of distributed systems are not. Scale for the load you have and the growth you expect, not for the load you hope to have someday. Second, these ideas are the foundation for the rest of the primer. Next come the trade-offs behind them, starting with performance vs scalability, then the building blocks one at a time.
Sessions are stored in each server's own memory, so when the load balancer sends a user to a different server, that server has no record of their session. The robust fix is to make the servers stateless: keep sessions in a shared store like Redis (or the database), so any server can handle any request. The quicker fix is sticky sessions, where the load balancer always sends a user to the same server. But that spreads load unevenly, and users on a server that dies still lose their sessions.
Vertical scaling is simpler, and for many workloads it goes surprisingly far, especially with careful tuning and a database that fits in RAM. But it has a ceiling (the biggest machine you can buy), costs climb steeply near the top end, and a single big machine is still a single point of failure. Traffic that spikes unpredictably, like Pokémon GO's launch at 50 times the plan, needs capacity you can add in minutes, which means scaling out. Many real systems do both: a few large database servers, and many small, disposable app servers.
Replication lag: the write went to the primary, but the read was served by a replica that hasn't caught up yet. Fix it by sending a user's reads to the primary for a short time after they write. A stale cache: the cached object wasn't deleted or updated when the recipe changed, so readers get the old copy until it expires. Fix it by deleting the object's cache key on every write. This is one reason caching whole objects is easier than caching query results.
Most activity stayed within one school, so the partition key matched how the data was used. Nearly every query touched only one small database, and adding a school meant adding a machine. It breaks down once many queries cross partitions. As Facebook opened to everyone, friendships crossed every school boundary, and a query like "friends at any school" would have to hit many databases and merge the results in application code. A good shard key keeps the data that's queried together on the same shard.
Save the original photo, put a "make thumbnails" job on a queue, and respond right away with a "processing" state. Workers take jobs from the queue and generate the thumbnails in the background. The new problem is a backlog: if uploads arrive faster than workers finish them, the queue keeps growing and thumbnails arrive later and later. Watch the queue length, add workers when it grows, and decide what to do when it's too long, such as rejecting or slowing new uploads (back pressure).
The originals this chapter is based on:
The real-world examples:
Previous: System design topics: start here · Next: Performance vs scalability