Web Search Engine Architecture

Web Search Engine Architecture An architecture diagram generated by Archify. Search Client · Browser / Mobile App · Architecture component Search Client Browser / Mobile App Query Gateway · Tokenizer & Dispatcher · Online Query Serving Cluster · Stateless Query Gateway Tokenizer & Dispatcher Stateless Query Cache · Redis In-Memory · Online Query Serving Cluster Query Cache Redis In-Memory Scoring Engine · BM25 + ML Ranker · Online Query Serving Cluster · Top-K Merge Scoring Engine BM25 + ML Ranker Top-K Merge Index Shard Fleet · Document-Partitioned · Online Query Serving Cluster Index Shard Fleet Document-Partitioned Document Store · Snippets & Meta (SSTable) · Online Query Serving Cluster Document Store Snippets & Meta (SSTable) Crawl Frontier · Politeness & Priority Queues · Offline Ingestion & Pipeline Cluster Crawl Frontier Politeness & Priority Queues Crawler Workers · HTML Fetch & SimHash · Offline Ingestion & Pipeline Cluster · Async Egress Crawler Workers HTML Fetch & SimHash Async Egress MapReduce Indexer · Inverted Index Builder · Offline Ingestion & Pipeline Cluster MapReduce Indexer Inverted Index Builder Web Page Corpus · Distributed Blob Store · Offline Ingestion & Pipeline Cluster Web Page Corpus Distributed Blob Store Search Query Check Hot Cache Dispatch Query Scatter-Gather Query Hydrate Top-K Snippets Poll Allowed URLs Store Raw HTML Stream Extracted Text Push SSTable Segments Online Query Serving Cluster Offline Ingestion & Pipeline Cluster Legend Backend Database Message bus External