{"id":"design-pastebin","title":"Design Pastebin.com (or Bit.ly)","sections":[{"id":"overview","title":"Overview","html":"

Note: This document links directly to relevant areas found in the system design topics to avoid duplication. Refer to the linked content for general talking points, tradeoffs, and alternatives.

\n

Design Bit.ly - is a similar question, except pastebin requires storing the paste contents instead of the original unshortened url.

","headings":[]},{"id":"step-1-outline-use-cases-and-constraints","title":"Step 1: Outline use cases and constraints","html":"

Step 1: Outline use cases and constraints

\n
\n

Gather requirements and scope the problem.\nAsk questions to clarify use cases and constraints.\nDiscuss assumptions.

\n
\n

Without an interviewer to address clarifying questions, we'll define some use cases and constraints.

\n

Use cases

\n

We'll scope the problem to handle only the following use cases

\n\n

Out of scope

\n\n

Constraints and assumptions

\n

State assumptions

\n\n

Calculate usage

\n

Clarify with your interviewer if you should run back-of-the-envelope usage calculations.

\n\n

Handy conversion guide:

\n","headings":[{"id":"use-cases","title":"Use cases","depth":3},{"id":"well-scope-the-problem-to-handle-only-the-following-use-cases","title":"We'll scope the problem to handle only the following use cases","depth":4},{"id":"out-of-scope","title":"Out of scope","depth":4},{"id":"constraints-and-assumptions","title":"Constraints and assumptions","depth":3},{"id":"state-assumptions","title":"State assumptions","depth":4},{"id":"calculate-usage","title":"Calculate usage","depth":4}]},{"id":"step-2-create-a-high-level-design","title":"Step 2: Create a high level design","html":"

Step 2: Create a high level design

\n
\n

Outline a high level design with all important components.

\n
\n

\"Diagram\"

","headings":[]},{"id":"step-3-design-core-components","title":"Step 3: Design core components","html":"

Step 3: Design core components

\n
\n

Dive into details for each core component.

\n
\n

Use case: User enters a block of text and gets a randomly generated link

\n

We could use a relational database as a large hash table, mapping the generated url to a file server and path containing the paste file.

\n

Instead of managing a file server, we could use a managed Object Store such as Amazon S3 or a NoSQL document store.

\n

An alternative to a relational database acting as a large hash table, we could use a NoSQL key-value store. We should discuss the tradeoffs between choosing SQL or NoSQL. The following discussion uses the relational database approach.

\n\n

Clarify with your interviewer how much code you are expected to write.

\n

The pastes table could have the following structure:

\n
shortlink char(7) NOT NULL\nexpiration_length_in_minutes int NOT NULL\ncreated_at datetime NOT NULL\npaste_path varchar(255) NOT NULL\nPRIMARY KEY(shortlink)
\n

Setting the primary key to be based on the shortlink column creates an index that the database uses to enforce uniqueness. We'll create an additional index on created_at to speed up lookups (log-time instead of scanning the entire table) and to keep the data in memory. Reading 1 MB sequentially from memory takes about 250 microseconds, while reading from SSD takes 4x and from disk takes 80x longer.<a href=https://github.com/donnemartin/system-design-primer#latency-numbers-every-programmer-should-know>1

\n

To generate the unique url, we could:

\n\n
def base_encode(num, base=62):\n    digits = []\n    while num > 0\n      remainder = modulo(num, base)\n      digits.push(remainder)\n      num = divide(num, base)\n    digits = digits.reverse
\n\n
url = base_encode(md5(ip_address+timestamp))[:URL_LENGTH]
\n

We'll use a public REST API:

\n
$ curl -X POST --data '{ \"expiration_length_in_minutes\": \"60\", \\\n    \"paste_contents\": \"Hello World!\" }' https://pastebin.com/api/v1/paste
\n

Response:

\n
{\n    \"shortlink\": \"foobar\"\n}
\n

For internal communications, we could use Remote Procedure Calls.

\n

Use case: User enters a paste's url and views the contents

\n\n

REST API:

\n
$ curl https://pastebin.com/api/v1/paste?shortlink=foobar
\n

Response:

\n
{\n    \"paste_contents\": \"Hello World\"\n    \"created_at\": \"YYYY-MM-DD HH:MM:SS\"\n    \"expiration_length_in_minutes\": \"60\"\n}
\n

Use case: Service tracks analytics of pages

\n

Since realtime analytics are not a requirement, we could simply MapReduce the Web Server logs to generate hit counts.

\n

Clarify with your interviewer how much code you are expected to write.

\n
class HitCounts(MRJob):\n\n    def extract_url(self, line):\n        \"\"\"Extract the generated url from the log line.\"\"\"\n        ...\n\n    def extract_year_month(self, line):\n        \"\"\"Return the year and month portions of the timestamp.\"\"\"\n        ...\n\n    def mapper(self, _, line):\n        \"\"\"Parse each log line, extract and transform relevant lines.\n\n        Emit key value pairs of the form:\n\n        (2016-01, url0), 1\n        (2016-01, url0), 1\n        (2016-01, url1), 1\n        \"\"\"\n        url = self.extract_url(line)\n        period = self.extract_year_month(line)\n        yield (period, url), 1\n\n    def reducer(self, key, values):\n        \"\"\"Sum values for each key.\n\n        (2016-01, url0), 2\n        (2016-01, url1), 1\n        \"\"\"\n        yield key, sum(values)
\n

Use case: Service deletes expired pastes

\n

To delete expired pastes, we could just scan the SQL Database for all entries whose expiration timestamp are older than the current timestamp. All expired entries would then be deleted (or marked as expired) from the table.

","headings":[{"id":"use-case-user-enters-a-block-of-text-and-gets-a-randomly-generated-link","title":"Use case: User enters a block of text and gets a randomly generated link","depth":3},{"id":"use-case-user-enters-a-pastes-url-and-views-the-contents","title":"Use case: User enters a paste's url and views the contents","depth":3},{"id":"use-case-service-tracks-analytics-of-pages","title":"Use case: Service tracks analytics of pages","depth":3},{"id":"use-case-service-deletes-expired-pastes","title":"Use case: Service deletes expired pastes","depth":3}]},{"id":"step-4-scale-the-design","title":"Step 4: Scale the design","html":"

Step 4: Scale the design

\n
\n

Identify and address bottlenecks, given the constraints.

\n
\n

\"Diagram\"

\n

Important: Do not simply jump right into the final design from the initial design!

\n

State you would do this iteratively: 1) Benchmark/Load Test, 2) Profile for bottlenecks 3) address bottlenecks while evaluating alternatives and trade-offs, and 4) repeat. See Design a system that scales to millions of users on AWS as a sample on how to iteratively scale the initial design.

\n

It's important to discuss what bottlenecks you might encounter with the initial design and how you might address each of them. For example, what issues are addressed by adding a Load Balancer with multiple Web Servers? CDN? Master-Slave Replicas? What are the alternatives and Trade-Offs for each?

\n

We'll introduce some components to complete the design and to address scalability issues. Internal load balancers are not shown to reduce clutter.

\n

To avoid repeating discussions, refer to the following system design topics for main talking points, tradeoffs, and alternatives:

\n\n

The Analytics Database could use a data warehousing solution such as Amazon Redshift or Google BigQuery.

\n

An Object Store such as Amazon S3 can comfortably handle the constraint of 12.7 GB of new content per month.

\n

To address the 40 average read requests per second (higher at peak), traffic for popular content should be handled by the Memory Cache instead of the database. The Memory Cache is also useful for handling the unevenly distributed traffic and traffic spikes. The SQL Read Replicas should be able to handle the cache misses, as long as the replicas are not bogged down with replicating writes.

\n

4 average paste writes per second (with higher at peak) should be do-able for a single SQL Write Master-Slave. Otherwise, we'll need to employ additional SQL scaling patterns:

\n\n

We should also consider moving some data to a NoSQL Database.

","headings":[]},{"id":"additional-talking-points","title":"Additional talking points","html":"

Additional talking points

\n
\n

Additional topics to dive into, depending on the problem scope and time remaining.

\n
\n

NoSQL

\n\n

Caching

\n\n

Asynchronism and microservices

\n\n

Communications

\n\n

Security

\n

Refer to the security section.

\n

Latency numbers

\n

See Latency numbers every programmer should know.

\n

Ongoing

\n","headings":[{"id":"nosql","title":"NoSQL","depth":4},{"id":"caching","title":"Caching","depth":3},{"id":"asynchronism-and-microservices","title":"Asynchronism and microservices","depth":3},{"id":"communications","title":"Communications","depth":3},{"id":"security","title":"Security","depth":3},{"id":"latency-numbers","title":"Latency numbers","depth":3},{"id":"ongoing","title":"Ongoing","depth":3}]}],"sourceFiles":[{"name":"pastebin.py","url":"https://github.com/donnemartin/system-design-primer/blob/master/solutions/system_design/pastebin/pastebin.py","html":"
# -*- coding: utf-8 -*-\n\nfrom mrjob.job import MRJob\n\n\nclass HitCounts(MRJob):\n\n    def extract_url(self, line):\n        \"\"\"Extract the generated url from the log line.\"\"\"\n        pass\n\n    def extract_year_month(self, line):\n        \"\"\"Return the year and month portions of the timestamp.\"\"\"\n        pass\n\n    def mapper(self, _, line):\n        \"\"\"Parse each log line, extract and transform relevant lines.\n\n        Emit key value pairs of the form:\n\n        (2016-01, url0), 1\n        (2016-01, url0), 1\n        (2016-01, url1), 1\n        \"\"\"\n        url = self.extract_url(line)\n        period = self.extract_year_month(line)\n        yield (period, url), 1\n\n    def reducer(self, key, values):\n        \"\"\"Sum values for each key.\n\n        (2016-01, url0), 2\n        (2016-01, url1), 1\n        \"\"\"\n        yield key, sum(values)\n\n    def steps(self):\n        \"\"\"Run the map and reduce steps.\"\"\"\n        return [\n            self.mr(mapper=self.mapper,\n                    reducer=self.reducer)\n        ]\n\n\nif __name__ == '__main__':\n    HitCounts.run()
"}]}