A web search engine takes a few words and, in a couple of hundred milliseconds, returns the ten most useful pages out of tens of billions. Behind that are three systems that run at very different speeds: a crawler that fetches the web continuously, an indexing pipeline that turns pages into a searchable structure, and a serving tier that answers a query by asking hundreds of machines at once.
\nThe primer's web crawler and query cache exercises cover two pieces. This chapter puts the whole thing together: how pages become an inverted index, how a crawler stays fast without hammering any one site, why large engines split the index by document rather than by term, and how BM25, PageRank and learned ranking combine to score results quickly. It builds on database 101, cache 101 and batch processing.
","headings":[]},{"id":"requirements-and-scale","title":"Requirements and scale","html":"Agree on scope first. A reasonable set:
\ndistributed consensus raft, return the 10 most relevant pages with a title, URL and a two-line snippet that highlights the query terms. Crawl the public web continuously, finding new pages and changes to known ones. Build and update the index from the crawl.Then size it. These numbers are assumptions for an interview, not figures from any real engine:
\nSee estimation for more on this kind of arithmetic.
","headings":[{"id":"requirements-and-scale","title":"Requirements and scale","depth":3}]},{"id":"high-level-architecture","title":"High-level architecture","html":"An interactive version of this diagram lets you pan, zoom and trace the request paths.
\nThe path of one query:
\nIf a shard replica is slow, the aggregator retries on another replica, or after a deadline returns results from the shards that answered. Missing 1 shard out of 500 drops 0.2% of the index for one query, which users rarely notice.
","headings":[{"id":"high-level-architecture","title":"High-level architecture","depth":3}]},{"id":"the-crawling-subsystem","title":"The crawling subsystem","html":"The crawler has two goals that pull against each other: freshness (fetch as much as possible, as often as useful) and politeness (never send one site more traffic than it can comfortably handle).
\nThe frontier is the set of URLs waiting to be fetched. The design popularized by the Mercator crawler, and described in detail in Introduction to Information Retrieval, has two layers of FIFO queues:
\nCrawl-delay in robots.txt if it sets one.Because each host is only ever in one back queue and one heap slot, at most one fetch per host is in flight. A slow site only slows itself; the fetchers meanwhile work through thousands of other hosts. That's how a crawler can make tens of thousands of requests a second overall while each site sees a trickle.
\nA few other pieces matter at scale. Fetchers keep their own DNS cache, because DNS lookups are slow and the crawler would otherwise flood resolvers. Each host's robots.txt is fetched, cached and checked before any URL from that host. The \"URL seen?\" check uses a large hash set or Bloom filter over normalized URLs, so the same page isn't queued twice from different links.
A large share of the web is near-duplicate: mirrors, syndicated articles, printer-friendly copies, pages that differ only in a date or a tracking parameter. Early studies of large crawls put it at around 30% of pages. Indexing them wastes space and fills result pages with copies.
\nAn exact hash like SHA-256 can't detect near-duplicates. Change one character and about half of the output bits flip, so the hash says nothing about how similar two pages are. SimHash is built so that similar documents get fingerprints that differ in only a few bits (a small Hamming distance):
\nTwo pages that share most of their features end up with the same sign in almost every position. Manku, Jain and Das Sarma at Google reported that for 8 billion pages, 64-bit fingerprints treated as near-duplicates at a Hamming distance of 3 or less worked well, and showed how to find all fingerprints within 3 bits of a new one quickly using several permuted, sorted tables.
","headings":[{"id":"the-crawling-subsystem","title":"The crawling subsystem","depth":3},{"id":"the-crawl-frontier-priority-and-politeness","title":"The crawl frontier: priority and politeness","depth":4},{"id":"near-duplicate-detection-with-simhash","title":"Near-duplicate detection with SimHash","depth":4}]},{"id":"the-inverted-index-pipeline","title":"The inverted index pipeline","html":"An inverted index maps each term to the list of documents that contain it, the opposite direction from a document that lists its words. It's what lets a shard find every page containing consensus without reading every page.
The dictionary holds every term with its document frequency (how many documents contain it) and where its posting list starts. The posting list holds one entry per document:
\n\"consensus\" df = 3\n doc 12 tf = 4 positions 4, 18, 92, 140\n doc 45 tf = 1 positions 12\n doc 89 tf = 8 positions 1, 9, 34, …Here tf is the term frequency: how many times the term occurs in that document.
\nWhen a fetched page reaches the indexer:
\nthe and of. Web engines mostly keep them, because queries such as to be or not to be or The Who need them, and compression makes their long posting lists affordable.running and runs both become run. Rules can't handle irregular forms, so ran stays ran; mapping it to run needs a lemmatizer with a dictionary. Engines often index both the original and the stemmed form so exact matches can still score higher.The index itself is built in batch: mappers emit (term, doc, positions) records, the shuffle groups them by term, and reducers write sorted posting lists. That's the original use case for MapReduce (see batch processing). To keep news fresh without rebuilding everything, engines also run a small, frequently updated index for recent pages, query it alongside the main one, and merge the results.
\nPositions make two things possible:
\n\"system design\", a document matches only if some position of design is exactly one more than a position of system.Uncompressed, the positions alone would be about 600 TB, as estimated above. Posting lists compress very well because they're sorted:
\n104, 108, 115, 120 are stored as 104, 4, 7, 5, the difference from the previous ID. Positions within a document are gap-encoded the same way. Gaps are small numbers, especially for common terms.Together these typically cut posting lists to a quarter or less of their uncompressed size, which is where the ~300 TB estimate comes from. For sets of document IDs used as filters (pages in a language, pages from a site), many engines use compressed bitmaps such as Roaring bitmaps instead.
","headings":[{"id":"the-inverted-index-pipeline","title":"The inverted index pipeline","depth":3},{"id":"document-processing","title":"Document processing","depth":4},{"id":"why-positions-are-stored","title":"Why positions are stored","depth":4},{"id":"compressing-posting-lists","title":"Compressing posting lists","depth":4}]},{"id":"partitioning-the-index-by-document-or-by-term","title":"Partitioning the index: by document or by term","html":"No single machine holds 300 TB of index and answers 100,000 queries a second, so the index is split. There are two ways to do it.
\nTerm partitioning (a global index). Each machine holds complete posting lists for a subset of terms, say every term from a to c on one node.
system design interview needs posting lists from three machines. Intersecting them means shipping long lists of document IDs across the network, and posting lists for common terms are huge.Document partitioning (a local index). Each machine holds a complete inverted index for a subset of documents, here 100 million pages per shard, assigned by a hash of the document ID or URL.
\n| Term partitioning | Document partitioning | |
|---|---|---|
| Machines per query | one per query term | all shards |
| Network traffic | posting lists | top-k results only |
| Load balance | skewed by popular terms | even |
| Effect of a lost node | some terms missing entirely | a fraction of documents missing |
| Adding documents | touches many nodes | touches one shard |
Large web engines use document partitioning in practice, adding replicas of each shard to handle query volume. Term partitioning shows up in research and in some specialized systems, but the network cost of multi-term queries makes it a poor fit here.
","headings":[{"id":"partitioning-the-index-by-document-or-by-term","title":"Partitioning the index: by document or by term","depth":3}]},{"id":"relevance-and-ranking","title":"Relevance and ranking","html":"Scoring is another funnel. Cheap scores run over many documents inside each shard, and expensive ones run over a few hundred at the aggregator.
\nThe starting idea is TF-IDF: a term matters more in a document the more often it appears there (term frequency) and the rarer it is across the collection (inverse document frequency):
\ntf-idf(t, D) = tf(t, D) × log(N ÷ df(t))N is the number of documents and df(t) is how many contain t. In a shard of 100 million pages, a term found in 100,000 of them gets log₁₀(10⁸ ÷ 10⁵) = 3, and one found in 10 million gets log₁₀(10) = 1. IDF grows with the logarithm of rarity, so a term 100 times rarer gets 3 times the weight, not 100 times.
\nTF-IDF has two flaws: a page that repeats a word 100 times scores 100 times higher, and long pages win simply by containing more words. BM25 (Okapi BM25, from Robertson and colleagues' probabilistic retrieval work) fixes both:
\nscore(D, Q) = sum over each query term q of\n IDF(q) × tf(q, D) × (k₁ + 1) ÷ ( tf(q, D) + k₁ × (1 − b + b × |D| ÷ avgdl) )\n\nIDF(q) = ln( 1 + (N − n(q) + 0.5) ÷ (n(q) + 0.5) )The IDF shown is the variant Lucene uses; the 1 + inside the log keeps IDF positive for terms that appear in more than half the documents.
PageRank, introduced by Brin and Page in 1998, treats a link as a vote, weighted by the importance of the page casting it. It models a \"random surfer\" who follows a random link on the current page with probability d, and otherwise jumps to a random page anywhere:
\nPR(u) = (1 − d) ÷ N + d × sum over pages v that link to u of PR(v) ÷ L(v)A page ranks high if many pages link to it, or if a few high-ranked pages do. PageRank is computed offline by power iteration: start with every page at 1 ÷ N, apply the formula to every page, and repeat until the values stop changing, typically after a few dozen rounds. Each round is a pass over the link graph, a natural fit for MapReduce or Spark. The result is stored as a static, per-document score that shards read at query time. Today it's one signal among hundreds, not the main ranking factor it was in 1998.
\nA shard with 100 million pages may have millions containing computer. Fully scoring every match for every query would waste most of the CPU, since only the top 1,000 matter.
WAND (\"weak AND\", Broder et al., 2003) skips documents that can't make the top k:
\nBlock-Max WAND (Ding and Suel, 2011) stores an upper bound for each block of about 128 postings as well as for the whole list. The block bounds are much tighter, so whole blocks can be skipped without being decompressed. Published experiments show several-fold speedups over plain WAND, and Lucene, which Elasticsearch and OpenSearch are built on, uses a block-max approach for top-k queries.
","headings":[{"id":"relevance-and-ranking","title":"Relevance and ranking","depth":3},{"id":"text-relevance-from-tf-idf-to-bm25","title":"Text relevance: from TF-IDF to BM25","depth":4},{"id":"link-authority-pagerank","title":"Link authority: PageRank","depth":4},{"id":"early-termination-with-wand","title":"Early termination with WAND","depth":4}]},{"id":"check-yourself","title":"Check yourself","html":"With term partitioning, a multi-word query like distributed system design needs posting lists from three machines, and intersecting them means sending long lists of document IDs over the network. Popular terms also turn their machines into hot spots.
With document partitioning, each shard has a complete index for its 100 million pages. The query goes to all shards at once, each intersects and scores locally, and each sends back only its top ~100 results, a few kilobytes. Load is spread evenly, and losing a shard costs a small slice of documents rather than whole terms. The price is that every query touches every shard and waits for the slowest one, which is why shards are replicated and the aggregator uses deadlines and hedged requests.
\nIt separates priority from politeness. Front queues order URLs by importance. Back queues each hold URLs from exactly one host, and a heap records the earliest time each host may be contacted again. A fetcher takes the host at the top of the heap, fetches one URL from that host's queue, then puts the host back with a new ready time, a multiple of how long the last fetch took and never less than the site's Crawl-delay.
Since each host appears once in the heap, only one fetch per host is in flight. While one site is in its cool-down, fetchers are busy with thousands of others, so total throughput stays high and no single site sees more than a trickle.
\nSHA-256 is designed so that changing one character flips about half the output bits. Two pages that differ only in a timestamp get unrelated hashes, so the hash tells you nothing about similarity.
\nSimHash hashes each feature (word or word sequence) to 64 bits and keeps 64 running totals: for each feature, add its weight where its hash has a 1 bit and subtract where it has a 0. The fingerprint's bit i is 1 if total i is positive. Pages that share most features end up with the same signs in almost every position, so their fingerprints differ in only a few bits. Google's crawler work treated 64-bit fingerprints within a Hamming distance of 3 as near-duplicates.
\nTerm frequency saturates. With k₁ = 1.2 a term's factor approaches k₁ + 1 = 2.2 no matter how many times it appears; at tf = 10 it's already about 1.96, so 200 repetitions add very little over 10. Length normalization (b) also lowers the score of a page bloated with extra words. And BM25 is only the first filter: the light and heavy rankers use link signals, spam classifiers and many other features that keyword stuffing doesn't improve.
\nAt 100,000 queries a second, 70,000 miss the cache. Each goes to all 500 shards, so that's 35 million shard queries a second. At an assumed 1,000 queries a second per replica, that's 35,000 shard servers, about 70 replicas per shard.
\nWith a 60% hit rate, 40,000 queries miss, so 20 million shard queries a second and about 20,000 servers. Because of the fan-out, every point of cache hit rate saves 500 shard queries per avoided query, which is why search engines invest heavily in the query cache.
\n