{"id":"design-query-cache","title":"Design a key-value cache to save the results of the most recent web server queries","sections":[{"id":"overview","title":"Overview","html":"

Note: This document links directly to relevant areas found in the system design topics to avoid duplication. Refer to the linked content for general talking points, tradeoffs, and alternatives.

","headings":[]},{"id":"step-1-outline-use-cases-and-constraints","title":"Step 1: Outline use cases and constraints","html":"

Step 1: Outline use cases and constraints

\n
\n

Gather requirements and scope the problem.\nAsk questions to clarify use cases and constraints.\nDiscuss assumptions.

\n
\n

Without an interviewer to address clarifying questions, we'll define some use cases and constraints.

\n

Use cases

\n

We'll scope the problem to handle only the following use cases

\n\n

Constraints and assumptions

\n

State assumptions

\n\n

Calculate usage

\n

Clarify with your interviewer if you should run back-of-the-envelope usage calculations.

\n\n

Handy conversion guide:

\n","headings":[{"id":"use-cases","title":"Use cases","depth":3},{"id":"well-scope-the-problem-to-handle-only-the-following-use-cases","title":"We'll scope the problem to handle only the following use cases","depth":4},{"id":"constraints-and-assumptions","title":"Constraints and assumptions","depth":3},{"id":"state-assumptions","title":"State assumptions","depth":4},{"id":"calculate-usage","title":"Calculate usage","depth":4}]},{"id":"step-2-create-a-high-level-design","title":"Step 2: Create a high level design","html":"

Step 2: Create a high level design

\n
\n

Outline a high level design with all important components.

\n
\n

\"Diagram\"

","headings":[]},{"id":"step-3-design-core-components","title":"Step 3: Design core components","html":"

Step 3: Design core components

\n
\n

Dive into details for each core component.

\n
\n

Use case: User sends a request resulting in a cache hit

\n

Popular queries can be served from a Memory Cache such as Redis or Memcached to reduce read latency and to avoid overloading the Reverse Index Service and Document Service. Reading 1 MB sequentially from memory takes about 250 microseconds, while reading from SSD takes 4x and from disk takes 80x longer.<a href=https://github.com/donnemartin/system-design-primer#latency-numbers-every-programmer-should-know>1

\n

Since the cache has limited capacity, we'll use a least recently used (LRU) approach to expire older entries.

\n\n

Cache implementation

\n

The cache can use a doubly-linked list: new items will be added to the head while items to expire will be removed from the tail. We'll use a hash table for fast lookups to each linked list node.

\n

Clarify with your interviewer how much code you are expected to write.

\n

Query API Server implementation:

\n
class QueryApi(object):\n\n    def __init__(self, memory_cache, reverse_index_service):\n        self.memory_cache = memory_cache\n        self.reverse_index_service = reverse_index_service\n\n    def parse_query(self, query):\n        \"\"\"Remove markup, break text into terms, deal with typos,\n        normalize capitalization, convert to use boolean operations.\n        \"\"\"\n        ...\n\n    def process_query(self, query):\n        query = self.parse_query(query)\n        results = self.memory_cache.get(query)\n        if results is None:\n            results = self.reverse_index_service.process_search(query)\n            self.memory_cache.set(query, results)\n        return results
\n

Node implementation:

\n
class Node(object):\n\n    def __init__(self, query, results):\n        self.query = query\n        self.results = results
\n

LinkedList implementation:

\n
class LinkedList(object):\n\n    def __init__(self):\n        self.head = None\n        self.tail = None\n\n    def move_to_front(self, node):\n        ...\n\n    def append_to_front(self, node):\n        ...\n\n    def remove_from_tail(self):\n        ...
\n

Cache implementation:

\n
class Cache(object):\n\n    def __init__(self, MAX_SIZE):\n        self.MAX_SIZE = MAX_SIZE\n        self.size = 0\n        self.lookup = {}  # key: query, value: node\n        self.linked_list = LinkedList()\n\n    def get(self, query)\n        \"\"\"Get the stored query result from the cache.\n\n        Accessing a node updates its position to the front of the LRU list.\n        \"\"\"\n        node = self.lookup[query]\n        if node is None:\n            return None\n        self.linked_list.move_to_front(node)\n        return node.results\n\n    def set(self, results, query):\n        \"\"\"Set the result for the given query key in the cache.\n\n        When updating an entry, updates its position to the front of the LRU list.\n        If the entry is new and the cache is at capacity, removes the oldest entry\n        before the new entry is added.\n        \"\"\"\n        node = self.lookup[query]\n        if node is not None:\n            # Key exists in cache, update the value\n            node.results = results\n            self.linked_list.move_to_front(node)\n        else:\n            # Key does not exist in cache\n            if self.size == self.MAX_SIZE:\n                # Remove the oldest entry from the linked list and lookup\n                self.lookup.pop(self.linked_list.tail.query, None)\n                self.linked_list.remove_from_tail()\n            else:\n                self.size += 1\n            # Add the new key and value\n            new_node = Node(query, results)\n            self.linked_list.append_to_front(new_node)\n            self.lookup[query] = new_node
\n

When to update the cache

\n

The cache should be updated when:

\n\n

The most straightforward way to handle these cases is to simply set a max time that a cached entry can stay in the cache before it is updated, usually referred to as time to live (TTL).

\n

Refer to When to update the cache for tradeoffs and alternatives. The approach above describes cache-aside.

","headings":[{"id":"use-case-user-sends-a-request-resulting-in-a-cache-hit","title":"Use case: User sends a request resulting in a cache hit","depth":3},{"id":"cache-implementation","title":"Cache implementation","depth":4},{"id":"when-to-update-the-cache","title":"When to update the cache","depth":4}]},{"id":"step-4-scale-the-design","title":"Step 4: Scale the design","html":"

Step 4: Scale the design

\n
\n

Identify and address bottlenecks, given the constraints.

\n
\n

\"Diagram\"

\n

Important: Do not simply jump right into the final design from the initial design!

\n

State you would 1) Benchmark/Load Test, 2) Profile for bottlenecks 3) address bottlenecks while evaluating alternatives and trade-offs, and 4) repeat. See Design a system that scales to millions of users on AWS as a sample on how to iteratively scale the initial design.

\n

It's important to discuss what bottlenecks you might encounter with the initial design and how you might address each of them. For example, what issues are addressed by adding a Load Balancer with multiple Web Servers? CDN? Master-Slave Replicas? What are the alternatives and Trade-Offs for each?

\n

We'll introduce some components to complete the design and to address scalability issues. Internal load balancers are not shown to reduce clutter.

\n

To avoid repeating discussions, refer to the following system design topics for main talking points, tradeoffs, and alternatives:

\n\n

Expanding the Memory Cache to many machines

\n

To handle the heavy request load and the large amount of memory needed, we'll scale horizontally. We have three main options on how to store the data on our Memory Cache cluster:

\n","headings":[{"id":"expanding-the-memory-cache-to-many-machines","title":"Expanding the Memory Cache to many machines","depth":3}]},{"id":"additional-talking-points","title":"Additional talking points","html":"

Additional talking points

\n
\n

Additional topics to dive into, depending on the problem scope and time remaining.

\n
\n

SQL scaling patterns

\n\n

NoSQL

\n\n

Caching

\n\n

Asynchronism and microservices

\n\n

Communications

\n\n

Security

\n

Refer to the security section.

\n

Latency numbers

\n

See Latency numbers every programmer should know.

\n

Ongoing

\n","headings":[{"id":"sql-scaling-patterns","title":"SQL scaling patterns","depth":3},{"id":"nosql","title":"NoSQL","depth":4},{"id":"caching","title":"Caching","depth":3},{"id":"asynchronism-and-microservices","title":"Asynchronism and microservices","depth":3},{"id":"communications","title":"Communications","depth":3},{"id":"security","title":"Security","depth":3},{"id":"latency-numbers","title":"Latency numbers","depth":3},{"id":"ongoing","title":"Ongoing","depth":3}]}],"sourceFiles":[{"name":"query_cache_snippets.py","url":"https://github.com/donnemartin/system-design-primer/blob/master/solutions/system_design/query_cache/query_cache_snippets.py","html":"
# -*- coding: utf-8 -*-\n\n\nclass QueryApi(object):\n\n    def __init__(self, memory_cache, reverse_index_cluster):\n        self.memory_cache = memory_cache\n        self.reverse_index_cluster = reverse_index_cluster\n\n    def parse_query(self, query):\n        \"\"\"Remove markup, break text into terms, deal with typos,\n        normalize capitalization, convert to use boolean operations.\n        \"\"\"\n        ...\n\n    def process_query(self, query):\n        query = self.parse_query(query)\n        results = self.memory_cache.get(query)\n        if results is None:\n            results = self.reverse_index_cluster.process_search(query)\n            self.memory_cache.set(query, results)\n        return results\n\n\nclass Node(object):\n\n    def __init__(self, query, results):\n        self.query = query\n        self.results = results\n\n\nclass LinkedList(object):\n\n    def __init__(self):\n        self.head = None\n        self.tail = None\n\n    def move_to_front(self, node):\n        ...\n\n    def append_to_front(self, node):\n        ...\n\n    def remove_from_tail(self):\n        ...\n\n\nclass Cache(object):\n\n    def __init__(self, MAX_SIZE):\n        self.MAX_SIZE = MAX_SIZE\n        self.size = 0\n        self.lookup = {}\n        self.linked_list = LinkedList()\n\n    def get(self, query):\n        \"\"\"Get the stored query result from the cache.\n\n        Accessing a node updates its position to the front of the LRU list.\n        \"\"\"\n        node = self.lookup[query]\n        if node is None:\n            return None\n        self.linked_list.move_to_front(node)\n        return node.results\n\n    def set(self, results, query):\n        \"\"\"Set the result for the given query key in the cache.\n\n        When updating an entry, updates its position to the front of the LRU list.\n        If the entry is new and the cache is at capacity, removes the oldest entry\n        before the new entry is added.\n        \"\"\"\n        node = self.map[query]\n        if node is not None:\n            # Key exists in cache, update the value\n            node.results = results\n            self.linked_list.move_to_front(node)\n        else:\n            # Key does not exist in cache\n            if self.size == self.MAX_SIZE:\n                # Remove the oldest entry from the linked list and lookup\n                self.lookup.pop(self.linked_list.tail.query, None)\n                self.linked_list.remove_from_tail()\n            else:\n                self.size += 1\n            # Add the new key and value\n            new_node = Node(query, results)\n            self.linked_list.append_to_front(new_node)\n            self.lookup[query] = new_node
"}]}