System Design Primer · llms.txt
Beyond the primer: design questions
Written for this site, not part of The System Design Primer. The sources it draws on are listed under Further reading.
A file sync service keeps a folder identical across a user's laptop, desktop and phone, and across everyone it's shared with. Save a file on one machine and, a few seconds later, it appears on the others. Users also expect older versions to be recoverable, edits made offline to sync when they reconnect, and nothing to be silently overwritten.
The question is a favorite because it separates two kinds of data with very different needs: small, constantly changing metadata (which files exist, where, at which version) that must be strongly consistent, and large, immutable content that just needs to be stored durably and moved efficiently. Most of the good ideas in the design come from keeping those two apart. It draws on the primer's database and consistency patterns chapters.
Scope:
These numbers are assumptions, rounded for easy arithmetic:
The estimate shows the shape: an exabyte-scale blob store with modest write rates, plus a metadata store that is far smaller but takes every commit and must never disagree with itself. Estimation has more on this kind of sizing.
Group files into namespaces. Each user's home folder is one namespace. Each shared folder is its own namespace, mounted into every member's tree at a path they choose. A namespace has an append-only journal: every change (create, modify, rename, delete) gets the next journal ID within that namespace.
| Table | Key | Holds |
|---|---|---|
journal | (ns_id, journal_id) | path, block list, size, rev, mtime, deleted flag |
files_latest | (ns_id, path) | the current entry for each path, for fast listing |
mounts | (user_id, ns_id) | where a shared namespace appears in the user's tree |
acl | (ns_id, user_id) | role: owner, editor or viewer |
blocks | hash | location in the block store, size, reference count |
A file's content is described by its block list: the ordered list of hashes of the blocks that make it up. The metadata never holds file bytes.
POST /commit {ns_id, path, block_list, base_rev}
-> {rev} | {missing: [hashes]} | 409 {current_rev}
PUT /blocks/{hash} <bytes>
GET /blocks/{hash} -> bytes
POST /changes {cursor} -> {entries[], cursor, has_more}
POST /changes/wait {cursor, timeout: 60} -> {changes: true|false}
POST /share {ns_id, user, role}
GET /revisions {ns_id, path} -> [{rev, block_list, mtime}]base_rev is the revision the client's edit started from; it's how the server detects conflicts. The cursor is opaque to the client. Internally it records the last journal ID the client has seen in each namespace it can access.
Saving a file on device A:
commit with the block list and the base_rev it edited from.base_rev against the current revision, appends a journal entry, and updates files_latest in one transaction. It returns the new rev.On device B, a changes/wait request has been open for up to 60 seconds. It returns changes: true; B calls changes with its cursor, gets the new entry, compares the block list with blocks it already has locally, downloads only the missing ones, writes the file to a temporary path, and renames it into place so no program ever sees a half-written file. Then it saves the new cursor.
This is the main design choice: metadata goes through a small, strongly consistent service; bytes go straight to a big, simple block store. They scale independently, and the block store never needs to understand files.
Files are split into blocks, and each block is stored under the hash of its contents, such as SHA-256. That's content addressing, and several properties fall out of it:
A sketch of the client side of a commit, with fixed-size blocks (it keeps the file in memory for clarity):
import hashlib
BLOCK = 4 * 1024 * 1024 # 4 MiB fixed-size blocks
def commit_file(path, ns_id, base_rev, server):
hashes, data = [], {}
with open(path, "rb") as f:
while chunk := f.read(BLOCK):
h = hashlib.sha256(chunk).hexdigest()
hashes.append(h)
data[h] = chunk
while True:
r = server.commit(ns_id, path, block_list=hashes, base_rev=base_rev)
if r.status == "missing":
for h in r.missing:
server.put_block(h, data[h])
elif r.status == "conflict":
return save_conflicted_copy(path, hashes)
else:
return r.revFixed-size blocks are simple and fast, and Dropbox has described using 4 MB blocks. Their weakness is insertions: add one byte at the start of a file and every block boundary shifts, so every hash changes and the whole file uploads again.
Content-defined chunking sets boundaries where a rolling hash of the last few dozen bytes matches a pattern, so boundaries move with the content. An insertion changes only the chunk it lands in. The Low-Bandwidth File System paper is the classic description of this approach.
| Fixed-size blocks | Content-defined chunks | |
|---|---|---|
| Boundary rule | Every N bytes | Where a rolling hash matches a pattern |
| Insert near the start | Every later block changes | Only nearby chunks change |
| CPU cost | Low | Higher (rolling hash over every byte) |
| Chunk sizes | Uniform | Variable, within min and max limits |
Dedup handles files that are copied, moved or restored. Other techniques cover edits:
A version is just an old journal entry with its block list. Because blocks are immutable and shared, keeping 30 days of history costs metadata rows plus only the blocks that actually changed. Restoring an old version is a new commit that points at the old block list.
Deleting blocks needs care. A block can be freed only when no current file, no retained version and no in-progress upload references it. Reference counting is fast but easy to get wrong under failures; a periodic mark-and-sweep over the metadata is slower but self-correcting. Either way, add a grace period before physically deleting anything.
The journal is what makes sync incremental. A client never asks "what does the whole tree look like?" except on first sync. It asks "what changed since my cursor?", and the answer is a range scan of journal IDs in each namespace it can see.
Notification works on top of that:
changes/wait) for desktop clients. The request blocks on the server until a namespace it covers changes or the timeout passes, then the client lists changes. The notification itself carries no data, so a missed or duplicated wake-up is harmless: the cursor is the source of truth. Dropbox's public API has the same shape with its list_folder/longpoll endpoint.The notification tier holds those 18 million waiting requests. Each server keeps a map from namespace to waiting connections and subscribes to commit events for those namespaces from a pub/sub system. See real-time communication for the trade-offs between long polling and persistent connections.
Two devices edit the same file while one is offline. Both started from rev 7.
base_rev = 7. That matches the current revision, so it becomes rev 8.base_rev = 7, but the current revision is now 8. The server rejects it with 409.report (Device B's conflicted copy 2026-10-12).docx, commits that as a new file, and downloads rev 8 under the original name.This is optimistic concurrency, a compare-and-set on rev. The service can't merge the two edits, because it doesn't understand the file format, so it never picks a loser silently. Both versions survive and a human decides. Folder operations have their own cases (one device renames a folder while another adds a file inside it, or deletes it), and sync engines handle each one explicitly. Dropbox's write-up of rewriting its sync engine is a good account of how hard these cases are.
Sharing a folder makes it its own namespace with an ACL, mounted in each member's tree. Every metadata call checks that the user has the right role on the namespace. Viewers get changes and GET /blocks; editors also get commit.
Two subtleties are worth raising:
GET /blocks/{hash} served any block to anyone who knew its hash, a leaked block list would leak the file. Check that the requester can read some file that references the block. With cross-user dedup, that check alone isn't enough: a client could commit a block list of hashes it never had, the server would find no blocks missing, and the client would then own a file that references them. So a commit must not reference a block the client hasn't uploaded or otherwise proved it holds.For very large sharing graphs, permission checks become a system of their own. Google's Zanzibar paper describes one built for exactly this kind of workload.
The two halves of the system need different guarantees, and that's what lets them use different stores:
base_rev check has to be an atomic compare-and-set, or two devices could both believe they wrote rev 8. A sharded relational database, sharded by ns_id so each namespace's journal lives on one shard, gives you transactions where you need them. Cross-namespace moves become the hard case, because they touch two shards.Almost nothing. A rename creates journal entries with new paths but the same block lists. Each other device sees the entries, checks which hashes it already has locally, finds all of them, and moves the files on disk. Only metadata travels.
As soon as the journal entry exists, other devices can see it and try to download the file. If some blocks don't exist yet, those devices would fail or, worse, write an incomplete file. Requiring every block to be durable before the metadata commit means anything visible in the journal is always fully readable. It also keeps the block store free of any notion of "pending" files.
Whoever commits first wins the revision check and becomes the current version. The second commit fails because its base_rev is stale, so that client saves its version as a conflicted copy beside the original and downloads the winner. Both see both files after the next sync. Nobody's work is lost, but someone has to merge by hand, which is why this approach suits arbitrary files and not real-time documents.
With fixed blocks, the insertion shifts every following byte, so every block from the edit to the end of the file gets a new hash: nearly 200 MB to upload. With content-defined chunking, boundaries are set by content, so they re-synchronize shortly after the insertion; only the one or two chunks around the new paragraph change and get uploaded. Delta sync with rsync-style checksums would be another way to recover most of the savings with fixed blocks.
Not if the ordering rule holds: a commit is accepted only once the blocks are durable, and a device that can't find a block in its local region falls back to the region that has it. Because blocks are immutable and named by hash, a replica either has exactly the right bytes or doesn't have the block at all; there's never a stale version. Only the metadata journal needs a single, strongly consistent order.
Previous: Design a video streaming platform · Next: About the primer