{"id":"real-world-architectures","title":"Real-world architectures","sections":[{"id":"overview","title":"Overview","html":"

The rest of this handbook teaches ideas one at a time: caching, replication, sharding, queues, consensus. Real systems use many of them at once, and the interesting part is how they fit together and what each team gave up to get what it needed. Reading how real systems were built is the best follow-up to the handbook, for three reasons:

\n\n

This chapter is a reading list with notes. It has three parts: the papers and talks behind well-known systems, write-ups of how particular companies built their products, and the engineering blogs where companies still publish this kind of material.

\n

Details such as version numbers, machine counts and product names go out of date quickly, so there is no point memorizing them. Read for the shape of the design, with four questions in mind:

\n\n

A worked example: reading the Dynamo paper

\n

Here is what that looks like on one paper. Dynamo: Amazon's Highly Available Key-value Store was presented at SOSP in October 2007. It describes the store that some of Amazon's core services used to keep data such as shopping carts.

\n

The problem. The paper opens with the shopping cart. Customers \"should be able to view and add items to their shopping cart even if disks are failing, network routes are flapping, or data centers are being destroyed by tornados.\" At Amazon's size, the authors write, small and large components fail continuously. So the store must keep accepting writes while parts of it are broken.

\n

The constraints. Many Amazon services, such as shopping carts, best seller lists, session management, sales rank and the product catalog, only need to look data up by its primary key. The paper says that storing their data in a relational database \"would lead to inefficiencies and limit scale and availability,\" so Dynamo offers only a primary-key interface. Amazon also measured its service-level agreements at the 99.9th percentile of latency, so that the slowest 0.1% of requests counted. The paper reports that its 99.9th-percentile latencies were around 200 ms, an order of magnitude higher than the averages.

\n

The trade-offs. To stay writable during failures, \"Dynamo sacrifices consistency under certain failure scenarios.\" Two copies of the same cart can be updated separately and drift apart. Dynamo keeps both versions and lets the application merge them when it next reads the data. The design pushes \"the complexity of conflict resolution to the reads in order to ensure that writes are never rejected.\" The cost shows up in the cart: an \"add to cart\" operation is never lost, \"however, deleted items can resurface.\"

\n

The techniques. The paper's Table 1 maps each problem to the technique it used. Every one of them has a section in this handbook:

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
ProblemDynamo's techniqueWhere the handbook covers it
Spreading keys over machinesConsistent hashing, with virtual nodesConsistent hashing
Staying writable when copies disagreeVector clocks, with the conflict resolved when the data is readVersioning
Machines that are down for a short timeSloppy quorum and hinted handoffSloppy quorum and hinted handoff
Machines that are gone for goodAnti-entropy using Merkle treesAnti-entropy with Merkle trees
Knowing which machines are aliveA gossip-based membership protocolFailure detection with gossip
\n

The paper also shows the tuning knobs from quorums: each key is stored on N machines, a read waits for R of them and a write waits for W. \"The common (N,R,W) configuration used by several instances of Dynamo is (3,2,2).\" Because R + W = 4 is greater than N = 3, a read and a write that both succeed normally overlap on at least one machine.

\n

What you take away. You don't need to remember any of Dynamo's numbers. The lesson is the pattern: when losing a write costs more than showing slightly stale data, accept every write and merge conflicts later. The CAP chapter tells the same story from the theory side.

\n

What changed later. Amazon's managed database service, DynamoDB, has a similar name but is a different system. Its own paper, Amazon DynamoDB: A Scalable, Predictably Performant, and Fully Managed NoSQL Database Service, appeared at USENIX ATC in July 2022. Reading the two papers side by side is a good exercise in how a design changes when it becomes a service that outside customers pay for.

","headings":[{"id":"a-worked-example-reading-the-dynamo-paper","title":"A worked example: reading the Dynamo paper","depth":4}]},{"id":"how-to-read-an-architecture-write-up","title":"How to read an architecture write-up","html":"

How to read an architecture write-up

\n

Use this checklist on any paper, talk or blog post in this chapter.

\n
    \n
  1. Find the problem first. Read the abstract and the introduction, then stop and write one sentence: who needed what, and why the existing tools weren't good enough.
    2. \n
  2. Write down the numbers that drove the design. Requests per second, reads versus writes, data size, latency targets, team size. These explain most of the choices.
    3. \n
  3. Sketch the architecture. Draw the boxes and arrows yourself before you look at the paper's figure. Mark where the data lives and which parts can fail on their own.
    4. \n
  4. For each component, ask what it solves. Then ask where it works, where it doesn't, and what would break first if traffic grew ten times.
    5. \n
  5. Name the trade-offs. Look for what they gave up: consistency for availability, flexibility for speed, generality for simplicity. Most papers say this outright in one or two sentences.
    6. \n
  6. Look for shared patterns and technologies. Compare it with other systems you have read about. Most designs reuse a small set of ideas, and spotting them is the real skill.
    7. \n
  7. Read the lessons learned. The \"experience\" or \"lessons\" section near the end is often the most useful part, because it is where the authors say what surprised them.
    8. \n
  8. Check the date. A 2008 post about a site running on a few dozen servers describes a different world from today's managed cloud services. The reasoning can still hold even when the products have changed.
    9. \n
  9. Skip the nitty-gritty on a first pass. Proofs, benchmark setups and configuration details can wait until you need them.
    10. \n
","headings":[{"id":"how-to-read-an-architecture-write-up","title":"How to read an architecture write-up","depth":3}]},{"id":"systems-and-papers","title":"Systems and papers","html":"

Systems and papers

\n

These are the systems most often cited in system design discussions. Where the original paper is freely available, it's linked. For open source projects, the link is an introductory slide deck, with the project's own site or original paper alongside it. The one-sentence descriptions come from the paper or project page.

\n

A worked example of the kind of diagram these write-ups contain: in a 2013 talk, Timelines at Scale, Twitter's Raffi Krikorian walked through how a tweet reaches people. High Scalability's summary of the talk reports 300,000 read requests a second against 6,000 writes, so Twitter did the work when a tweet was written. A \"fanout\" step copied each new tweet's ID into every follower's home timeline, kept in a Redis cluster with up to 800 entries per timeline and three copies of each. A separate path fed the tweet to an \"Ingester\", which indexed it in Earlybird, Twitter's search index, and a service called Blender built search results from it. The tweet also triggered push notifications.

\n
Write API; Write API -> Ingester; Write API -> Fanout; Write API -> Push notifications; Ingester -> Earlybird search index; Fanout -> Redis timeline cache (tweet IDs); Earlybird search index -> Blender; Redis timeline cache -> Timeline service; Blender -> Search results; Timeline service -> Home timeline\">
New tweetWrite APIIngesterFanoutPushnotificationsEarlybirdsearch indexRedistimeline cacheBlenderTimelineserviceSearch resultsHome timelinetweet IDs
How a tweet reached readers in 2013, after Twitter's \"Timelines at Scale\" talk
\n

Fan-out on write has a weak spot: accounts with tens of millions of followers. The summary reports that a tweet could take up to five minutes to reach all 31 million of Lady Gaga's followers, against a goal of five seconds. For accounts like that, Twitter was moving to skip the fan-out and merge their tweets into each reader's timeline at read time. Design the Twitter timeline and search builds the same design step by step.

\n

Data processing

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
SystemWhat it isReadHandbook chapters
MapReduce (Google)A programming model for processing large data sets: you write a map function and a reduce function, and the system runs them in parallel on thousands of machines and handles machine failures for you.Paper, OSDI 2004Batch processing
SparkA multi-language engine for data engineering, data science and machine learning on one machine or a cluster. It started as a research project at UC Berkeley's AMPLab in 2009 and is now an Apache project. Its core idea is keeping data in memory across steps.Slides: Apache Spark Architecture · RDD paper, NSDI 2012 · spark.apache.orgBatch processing
StormA distributed realtime computation system that processes unbounded streams of data, \"doing for realtime processing what Hadoop did for batch processing.\" It was built at BackType, which Twitter bought in July 2011, and Twitter open sourced it that September.Slides: Introduction to Storm · storm.apache.orgBatch processing, Event streaming
\n

Data stores

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
SystemWhat it isReadHandbook chapters
Bigtable (Google)A distributed storage system for structured data, built to scale to petabytes across thousands of commodity servers. The paper says more than sixty Google products used it, including web indexing, Google Earth and Google Finance.Paper, OSDI 2006Wide column stores
HBaseThe Hadoop database: an open source store modeled after Bigtable, running on top of Hadoop and HDFS, with random, realtime reads and writes.Slides: Intro to HBase · hbase.apache.orgWide column stores
CassandraA distributed NoSQL database with a partitioned wide-column model and eventual consistency. It was first designed at Facebook, combining Dynamo's distribution and replication techniques with Bigtable's data and storage model.Slides: Cassandra Introduction & Features · Paper, LADIS 2009Wide column stores, Consistent hashing
Dynamo (Amazon)A highly available key-value store that gives up consistency in some failure cases so that writes are never rejected. (DynamoDB, Amazon's later managed database, is a separate system with its own 2022 paper.)Dynamo paper, SOSP 2007 · DynamoDB paper, USENIX ATC 2022Design a key-value store, CAP and consistency
MongoDBA document database: each record is a document of field and value pairs, similar to a JSON object, and fields can hold other documents and arrays.Slides: Introduction to MongoDB · What is MongoDB?Document stores
Spanner (Google)Google's \"scalable, multi-version, globally-distributed, and synchronously-replicated database,\" the first to support consistent distributed transactions at global scale. It relies on a time API that exposes how uncertain the clock is.Paper, OSDI 2012CAP and consistency, Distributed transactions
MemcachedA free, open source, distributed in-memory key-value cache for small chunks of data, such as the results of database calls, API calls or page rendering. It speeds up web applications by taking load off the database.Slides: Introduction to memcached · memcached.orgCaching
RedisAn in-memory \"data structure server\" with native types such as strings, hashes, lists, sets and sorted sets. Unlike Memcached it can persist data to disk, either as point-in-time snapshots (RDB) or as a log of every write (AOF).Slides: Introduction to Redis · Redis data types · PersistenceCaching, Key-value stores
\n

File systems

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
SystemWhat it isReadHandbook chapters
Google File System (GFS)A scalable distributed file system for large, data-intensive applications, built to tolerate faults on cheap commodity hardware. A single master holds the metadata while many chunkservers hold the data.Paper, SOSP 2003Batch processing, Availability
Hadoop Distributed File System (HDFS)The open source counterpart to GFS. A single NameNode manages the file system's namespace, and DataNodes store the blocks that files are split into. It was originally built for the Apache Nutch web search engine.HDFS ArchitectureBatch processing
\n

Coordination and other infrastructure

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
SystemWhat it isReadHandbook chapters
Chubby (Google)A lock service for loosely coupled distributed systems. It offers coarse-grained locks and a small amount of reliable storage, through an interface much like a file system, and favors availability and reliability over speed. Paxos sits at its core.Paper, OSDI 2006Consensus
ZooKeeperA distributed, open source coordination service. It offers simple building blocks, arranged like a file system's directory tree, that applications use for synchronization, configuration, groups and naming. It came out of Yahoo.Slides: Introduction to Apache ZooKeeper · Paper, USENIX ATC 2010 · zookeeper.apache.orgConsensus
Dapper (Google)Google's tracing system for following one request through many services. It keeps overhead low by sampling requests and by adding instrumentation to a small number of shared libraries.Paper, Google technical report 2010Observability
KafkaA distributed messaging system built at LinkedIn to collect and deliver large volumes of log data with low latency. Today it describes itself as an event streaming platform.Slides: Introduction and Overview of Apache Kafka (TriHUG, July 2013) · Paper, NetDB 2011 · kafka.apache.orgEvent streaming, Queues and async
\n

Each paper describes its system as it was when the paper was published, often more than a decade ago. The systems have changed since, and some have been replaced. Read the papers for the reasoning behind each design, which ages much more slowly than the details.

","headings":[{"id":"systems-and-papers","title":"Systems and papers","depth":3},{"id":"data-processing","title":"Data processing","depth":4},{"id":"data-stores","title":"Data stores","depth":4},{"id":"file-systems","title":"File systems","depth":4},{"id":"coordination-and-other-infrastructure","title":"Coordination and other infrastructure","depth":4}]},{"id":"company-architectures","title":"Company architectures","html":"

Company architectures

\n

These write-ups describe how particular companies built their products. Most are posts on High Scalability, a blog that summarized conference talks and engineering posts for many years. Many are more than ten years old, so the year matters: read them for how the team reasoned under its constraints, and assume the specific tools have changed since. The year is the post's publication date, or the date of the talk where that's known.

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
CompanyWrite-upYearFormat
AmazonAmazon Architecture2007Article
CinchcastProducing 1,500 hours of audio every day2012Article
DataSiftRealtime datamining at 120,000 tweets per second2011Article
DropboxHow we've scaled Dropbox, Kevin Modzelewski at Stanford2012Video
ESPNOperating at 100,000 duh nuh nuhs per second2013Article
FacebookScaling Memcache at Facebook (NSDI)2013Paper
FacebookTAO: Facebook's distributed data store for the social graph (USENIX ATC)2013Paper
FacebookFinding a needle in Haystack: Facebook's photo storage (OSDI)2010Paper
FacebookHow Facebook Live streams to 800,000 simultaneous viewers2016Article
FlickrFlickr Architecture2007Article
GoogleGoogle Architecture2008Article
Instagram14 million users, terabytes of photos2011Article
InstagramWhat powers Instagram: hundreds of instances, dozens of technologies2011Article
Justin.tvJustin.tv's live video broadcasting architecture2010Article
MailboxFrom 0 to one million users in 6 weeks2013Article
NetflixA 360 degree view of the entire Netflix stack2015Article
NetflixNetflix: what happens when you press play?2017Article
PinterestFrom 0 to 10s of billions of page views a month2013Article
Pinterest18 million visitors, 10x growth, 12 employees2012Article
Playfish50 million monthly users and growing2010Article
PlentyOfFishPlentyOfFish Architecture2009Article
SalesforceHow they handle 1.3 billion transactions a day2013Article
Stack OverflowStack Overflow Architecture2009Article
TripAdvisor40M visitors, 200M dynamic page views, 30TB data2011Article
Tumblr15 billion page views a month2012Article
TwitterMaking Twitter 10000 percent faster2009Article
TwitterStoring 250 million tweets a day using MySQL2011Article
Twitter150M active users, 300K QPS, a 22 MB/s firehose2013Article
TwitterTimelines at scale, Raffi Krikorian on InfoQ2013Video
TwitterBig and small data at Twitter, Jeremy Cole at the O'Reilly MySQL Conference2011Video
TwitterOperations at Twitter: scaling beyond 100 million users, John Adams at LISA '102010Video
TwitterHow Twitter handles 3,000 images per second2016Article
UberHow Uber scales their real-time market platform2015Article
UberLessons learned from scaling Uber to 2000 engineers, 1000 services, and 8000 Git repositories2016Article
WhatsAppThe WhatsApp architecture Facebook bought for $19 billion2014Article
YouTubeYouTube Scalability, Cuong Do at a Google Tech Talk (uploaded in 2012)2007Video
YouTubeYouTube Architecture2008Article
\n

Some pairs are worth reading together, because each one shows a design changing as the company grew:

\n","headings":[{"id":"company-architectures","title":"Company architectures","depth":3}]},{"id":"company-engineering-blogs","title":"Company engineering blogs","html":"

Company engineering blogs

\n

If you are interviewing at a company, read a few recent posts from its engineering blog. The questions you get may come from the same domain, and the posts show which problems the company's engineers actually work on.

\n

Many of these blogs have moved since they were first started, and a few have closed. The links below are the current addresses, checked in October 2026. Where a blog has closed, the link is to an archived copy on the Internet Archive's Wayback Machine.

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
CompanyBlogNotes
AirbnbThe Airbnb Tech BlogMoved to Medium from its old address, nerds.airbnb.com.
AtlassianHow we buildThe old developer blog now redirects to the general \"Inside Atlassian\" blog. Archived copy of the developer blog (2019).
Amazon Web ServicesAWS News BlogProduct launches and how AWS services work.
Bitlyword: the Bitly engineering blogStill online, but the last post is from November 2017.
BoxBox Blog: Engineering
ClouderaCloudera BlogThe old developer blog address redirects here.
DropboxDropbox Tech BlogMoved from tech.dropbox.com.
eBayeBay Innovation storiesThe old eBay Tech Blog address redirects here.
EtsyCode as CraftMoved from codeascraft.com.
EvernoteEvernote Tech Blog (archived, 2017)Closed. The address now redirects to Evernote's product blog.
Facebook (Meta)Engineering at MetaReplaces the old Facebook Engineering page.
Flickrcode.flickr.netStill online. The latest post is from February 2022.
FoursquareEngineering posts on Foursquare's Mediumengineering.foursquare.com now redirects to Foursquare's marketing pages. The latest engineering post is from October 2022.
GitHubThe GitHub Blog: Engineering
GoogleGoogle Research BlogMoved from googleresearch.blogspot.com.
GrouponGroupon Product and EngineeringThe latest post is from June 2023.
HerokuHeroku Blog: Engineering
High ScalabilityHigh ScalabilityNot a company blog: summaries of how many companies built their systems, including most of the write-ups above.
HubSpotHubSpot Product Team Blog: Engineering
InstagramInstagram on Engineering at MetaInstagram moved its blog here in 2022. The original Tumblr blog is still online, with posts up to 2015.
IntelIntel Software Blogs (archived, 2015)Closed. The old address no longer loads.
Jane StreetJane Street Tech BlogMuch of it is about OCaml, the language Jane Street uses.
LinkedInLinkedIn Engineering Blog
MicrosoftEngineering@MicrosoftThe old engineering.microsoft.com site closed and now redirects to Azure DevOps documentation. Archived copy (2018).
MicrosoftMicrosoft for Python Developers BlogFormerly \"Python Engineering at Microsoft\" on MSDN.
NetflixNetflix TechBlogMoved from techblog.netflix.com.
PayPalPayPal Developer Community BlogPayPal's Medium blog now points here too.
PinterestPinterest Engineering Blog
QuoraQuora EngineeringThe address loads behind a browser check, so it couldn't be checked automatically.
RedditReddit Inc blog and r/RedditEngredditblog.com redirects to the company blog. Engineering posts go to the r/RedditEng community.
SalesforceSalesforce Engineering Blog
SlackEngineering at Slack
SpotifySpotify EngineeringThe old address, labs.spotify.com, now hosts Spotify's experimental products instead of the blog.
StripeStripe Dev Blog: Engineeringstripe.com/blog/engineering redirects here.
TwilioTwilio BlogThe old engineering address redirects to the main blog.
Twitter (X)X EngineeringMoved from blog.twitter.com. The page blocks automated checks, so here is an archived copy from September 2026.
UberUber Blog: EngineeringMoved from eng.uber.com.
YahooYahoo EngineeringStill online, but the last post is from March 2019.
YelpYelp Engineering and Product Blog
ZyngaZynga Engineering BlogPosts are infrequent.
\n

For many more, the community-maintained engineering-blogs list on GitHub collects hundreds of company and personal engineering blogs. If you know a company blog that's missing, add it there.

\n

Further reading

\n","headings":[{"id":"company-engineering-blogs","title":"Company engineering blogs","depth":3},{"id":"further-reading","title":"Further reading","depth":3}]}]}