{"id":"design-web-crawler","title":"Design a web crawler","sections":[{"id":"overview","title":"Overview","html":"

Note: This document links directly to relevant areas found in the system design topics to avoid duplication. Refer to the linked content for general talking points, tradeoffs, and alternatives.

","headings":[]},{"id":"step-1-outline-use-cases-and-constraints","title":"Step 1: Outline use cases and constraints","html":"

Step 1: Outline use cases and constraints

\n
\n

Gather requirements and scope the problem.\nAsk questions to clarify use cases and constraints.\nDiscuss assumptions.

\n
\n

Without an interviewer to address clarifying questions, we'll define some use cases and constraints.

\n

Use cases

\n

We'll scope the problem to handle only the following use cases

\n\n

Out of scope

\n\n

Constraints and assumptions

\n

State assumptions

\n\n

Exercise the use of more traditional systems - don't use existing systems such as solr or nutch.

\n

Calculate usage

\n

Clarify with your interviewer if you should run back-of-the-envelope usage calculations.

\n\n

Handy conversion guide:

\n","headings":[{"id":"use-cases","title":"Use cases","depth":3},{"id":"well-scope-the-problem-to-handle-only-the-following-use-cases","title":"We'll scope the problem to handle only the following use cases","depth":4},{"id":"out-of-scope","title":"Out of scope","depth":4},{"id":"constraints-and-assumptions","title":"Constraints and assumptions","depth":3},{"id":"state-assumptions","title":"State assumptions","depth":4},{"id":"calculate-usage","title":"Calculate usage","depth":4}]},{"id":"step-2-create-a-high-level-design","title":"Step 2: Create a high level design","html":"

Step 2: Create a high level design

\n
\n

Outline a high level design with all important components.

\n
\n

\"Diagram\"

","headings":[]},{"id":"step-3-design-core-components","title":"Step 3: Design core components","html":"

Step 3: Design core components

\n
\n

Dive into details for each core component.

\n
\n

Use case: Service crawls a list of urls

\n

We'll assume we have an initial list of links_to_crawl ranked initially based on overall site popularity. If this is not a reasonable assumption, we can seed the crawler with popular sites that link to outside content such as Yahoo, DMOZ, etc.

\n

We'll use a table crawled_links to store processed links and their page signatures.

\n

We could store links_to_crawl and crawled_links in a key-value NoSQL Database. For the ranked links in links_to_crawl, we could use Redis with sorted sets to maintain a ranking of page links. We should discuss the use cases and tradeoffs between choosing SQL or NoSQL.

\n\n

Clarify with your interviewer how much code you are expected to write.

\n

PagesDataStore is an abstraction within the Crawler Service that uses the NoSQL Database:

\n
class PagesDataStore(object):\n\n    def __init__(self, db);\n        self.db = db\n        ...\n\n    def add_link_to_crawl(self, url):\n        \"\"\"Add the given link to `links_to_crawl`.\"\"\"\n        ...\n\n    def remove_link_to_crawl(self, url):\n        \"\"\"Remove the given link from `links_to_crawl`.\"\"\"\n        ...\n\n    def reduce_priority_link_to_crawl(self, url)\n        \"\"\"Reduce the priority of a link in `links_to_crawl` to avoid cycles.\"\"\"\n        ...\n\n    def extract_max_priority_page(self):\n        \"\"\"Return the highest priority link in `links_to_crawl`.\"\"\"\n        ...\n\n    def insert_crawled_link(self, url, signature):\n        \"\"\"Add the given link to `crawled_links`.\"\"\"\n        ...\n\n    def crawled_similar(self, signature):\n        \"\"\"Determine if we've already crawled a page matching the given signature\"\"\"\n        ...
\n

Page is an abstraction within the Crawler Service that encapsulates a page, its contents, child urls, and signature:

\n
class Page(object):\n\n    def __init__(self, url, contents, child_urls, signature):\n        self.url = url\n        self.contents = contents\n        self.child_urls = child_urls\n        self.signature = signature
\n

Crawler is the main class within Crawler Service, composed of Page and PagesDataStore.

\n
class Crawler(object):\n\n    def __init__(self, data_store, reverse_index_queue, doc_index_queue):\n        self.data_store = data_store\n        self.reverse_index_queue = reverse_index_queue\n        self.doc_index_queue = doc_index_queue\n\n    def create_signature(self, page):\n        \"\"\"Create signature based on url and contents.\"\"\"\n        ...\n\n    def crawl_page(self, page):\n        for url in page.child_urls:\n            self.data_store.add_link_to_crawl(url)\n        page.signature = self.create_signature(page)\n        self.data_store.remove_link_to_crawl(page.url)\n        self.data_store.insert_crawled_link(page.url, page.signature)\n\n    def crawl(self):\n        while True:\n            page = self.data_store.extract_max_priority_page()\n            if page is None:\n                break\n            if self.data_store.crawled_similar(page.signature):\n                self.data_store.reduce_priority_link_to_crawl(page.url)\n            else:\n                self.crawl_page(page)
\n

Handling duplicates

\n

We need to be careful the web crawler doesn't get stuck in an infinite loop, which happens when the graph contains a cycle.

\n

Clarify with your interviewer how much code you are expected to write.

\n

We'll want to remove duplicate urls:

\n\n
class RemoveDuplicateUrls(MRJob):\n\n    def mapper(self, _, line):\n        yield line, 1\n\n    def reducer(self, key, values):\n        total = sum(values)\n        if total == 1:\n            yield key, total
\n

Detecting duplicate content is more complex. We could generate a signature based on the contents of the page and compare those two signatures for similarity. Some potential algorithms are Jaccard index and cosine similarity.

\n

Determining when to update the crawl results

\n

Pages need to be crawled regularly to ensure freshness. Crawl results could have a timestamp field that indicates the last time a page was crawled. After a default time period, say one week, all pages should be refreshed. Frequently updated or more popular sites could be refreshed in shorter intervals.

\n

Although we won't dive into details on analytics, we could do some data mining to determine the mean time before a particular page is updated, and use that statistic to determine how often to re-crawl the page.

\n

We might also choose to support a Robots.txt file that gives webmasters control of crawl frequency.

\n

Use case: User inputs a search term and sees a list of relevant pages with titles and snippets

\n\n

We'll use a public REST API:

\n
$ curl https://search.com/api/v1/search?query=hello+world
\n

Response:

\n
{\n    \"title\": \"foo's title\",\n    \"snippet\": \"foo's snippet\",\n    \"link\": \"https://foo.com\",\n},\n{\n    \"title\": \"bar's title\",\n    \"snippet\": \"bar's snippet\",\n    \"link\": \"https://bar.com\",\n},\n{\n    \"title\": \"baz's title\",\n    \"snippet\": \"baz's snippet\",\n    \"link\": \"https://baz.com\",\n},
\n

For internal communications, we could use Remote Procedure Calls.

","headings":[{"id":"use-case-service-crawls-a-list-of-urls","title":"Use case: Service crawls a list of urls","depth":3},{"id":"handling-duplicates","title":"Handling duplicates","depth":3},{"id":"determining-when-to-update-the-crawl-results","title":"Determining when to update the crawl results","depth":3},{"id":"use-case-user-inputs-a-search-term-and-sees-a-list-of-relevant-pages-with-titles-and-snippets","title":"Use case: User inputs a search term and sees a list of relevant pages with titles and snippets","depth":3}]},{"id":"step-4-scale-the-design","title":"Step 4: Scale the design","html":"

Step 4: Scale the design

\n
\n

Identify and address bottlenecks, given the constraints.

\n
\n

\"Diagram\"

\n

Important: Do not simply jump right into the final design from the initial design!

\n

State you would 1) Benchmark/Load Test, 2) Profile for bottlenecks 3) address bottlenecks while evaluating alternatives and trade-offs, and 4) repeat. See Design a system that scales to millions of users on AWS as a sample on how to iteratively scale the initial design.

\n

It's important to discuss what bottlenecks you might encounter with the initial design and how you might address each of them. For example, what issues are addressed by adding a Load Balancer with multiple Web Servers? CDN? Master-Slave Replicas? What are the alternatives and Trade-Offs for each?

\n

We'll introduce some components to complete the design and to address scalability issues. Internal load balancers are not shown to reduce clutter.

\n

To avoid repeating discussions, refer to the following system design topics for main talking points, tradeoffs, and alternatives:

\n\n

Some searches are very popular, while others are only executed once. Popular queries can be served from a Memory Cache such as Redis or Memcached to reduce response times and to avoid overloading the Reverse Index Service and Document Service. The Memory Cache is also useful for handling the unevenly distributed traffic and traffic spikes. Reading 1 MB sequentially from memory takes about 250 microseconds, while reading from SSD takes 4x and from disk takes 80x longer.<a href=https://github.com/donnemartin/system-design-primer#latency-numbers-every-programmer-should-know>1

\n

Below are a few other optimizations to the Crawling Service:

\n","headings":[]},{"id":"additional-talking-points","title":"Additional talking points","html":"

Additional talking points

\n
\n

Additional topics to dive into, depending on the problem scope and time remaining.

\n
\n

SQL scaling patterns

\n\n

NoSQL

\n\n

Caching

\n\n

Asynchronism and microservices

\n\n

Communications

\n\n

Security

\n

Refer to the security section.

\n

Latency numbers

\n

See Latency numbers every programmer should know.

\n

Ongoing

\n","headings":[{"id":"sql-scaling-patterns","title":"SQL scaling patterns","depth":3},{"id":"nosql","title":"NoSQL","depth":4},{"id":"caching","title":"Caching","depth":3},{"id":"asynchronism-and-microservices","title":"Asynchronism and microservices","depth":3},{"id":"communications","title":"Communications","depth":3},{"id":"security","title":"Security","depth":3},{"id":"latency-numbers","title":"Latency numbers","depth":3},{"id":"ongoing","title":"Ongoing","depth":3}]}],"sourceFiles":[{"name":"web_crawler_mapreduce.py","url":"https://github.com/donnemartin/system-design-primer/blob/master/solutions/system_design/web_crawler/web_crawler_mapreduce.py","html":"
# -*- coding: utf-8 -*-\n\nfrom mrjob.job import MRJob\n\n\nclass RemoveDuplicateUrls(MRJob):\n\n    def mapper(self, _, line):\n        yield line, 1\n\n    def reducer(self, key, values):\n        total = sum(values)\n        if total == 1:\n            yield key, total\n\n    def steps(self):\n        \"\"\"Run the map and reduce steps.\"\"\"\n        return [\n            self.mr(mapper=self.mapper,\n                    reducer=self.reducer)\n        ]\n\n\nif __name__ == '__main__':\n    RemoveDuplicateUrls.run()
"},{"name":"web_crawler_snippets.py","url":"https://github.com/donnemartin/system-design-primer/blob/master/solutions/system_design/web_crawler/web_crawler_snippets.py","html":"
# -*- coding: utf-8 -*-\n\n\nclass PagesDataStore(object):\n\n    def __init__(self, db):\n        self.db = db\n        pass\n\n    def add_link_to_crawl(self, url):\n        \"\"\"Add the given link to `links_to_crawl`.\"\"\"\n        pass\n\n    def remove_link_to_crawl(self, url):\n        \"\"\"Remove the given link from `links_to_crawl`.\"\"\"\n        pass\n\n    def reduce_priority_link_to_crawl(self, url):\n        \"\"\"Reduce the priority of a link in `links_to_crawl` to avoid cycles.\"\"\"\n        pass\n\n    def extract_max_priority_page(self):\n        \"\"\"Return the highest priority link in `links_to_crawl`.\"\"\"\n        pass\n\n    def insert_crawled_link(self, url, signature):\n        \"\"\"Add the given link to `crawled_links`.\"\"\"\n        pass\n\n    def crawled_similar(self, signature):\n        \"\"\"Determine if we've already crawled a page matching the given signature\"\"\"\n        pass\n\n\nclass Page(object):\n\n    def __init__(self, url, contents, child_urls):\n        self.url = url\n        self.contents = contents\n        self.child_urls = child_urls\n        self.signature = self.create_signature()\n\n    def create_signature(self):\n        # Create signature based on url and contents\n        pass\n\n\nclass Crawler(object):\n\n    def __init__(self, pages, data_store, reverse_index_queue, doc_index_queue):\n        self.pages = pages\n        self.data_store = data_store\n        self.reverse_index_queue = reverse_index_queue\n        self.doc_index_queue = doc_index_queue\n\n    def crawl_page(self, page):\n        for url in page.child_urls:\n            self.data_store.add_link_to_crawl(url)\n        self.reverse_index_queue.generate(page)\n        self.doc_index_queue.generate(page)\n        self.data_store.remove_link_to_crawl(page.url)\n        self.data_store.insert_crawled_link(page.url, page.signature)\n\n    def crawl(self):\n        while True:\n            page = self.data_store.extract_max_priority_page()\n            if page is None:\n                break\n            if self.data_store.crawled_similar(page.signature):\n                self.data_store.reduce_priority_link_to_crawl(page.url)\n            else:\n                self.crawl_page(page)\n            page = self.data_store.extract_max_priority_page()
"}]}