System Design Handbook · llms.txt
A developer creates an index called support-docs with 1,536 dimensions and cosine similarity, then upserts 200,000 vectors with metadata such as {"lang": "de", "product": "billing"}. Their app then asks for the 10 vectors nearest to a query among German billing documents, and expects an answer in milliseconds that includes the document written a second ago. The same service hosts a retailer with two billion image vectors and 50,000 customers whose indexes nobody queried this month. Pinecone, Weaviate Cloud, Qdrant Cloud and Turbopuffer sell services of this kind.
Embeddings and vector search explains the indexes used here, and the storage engine borrows from LSM-trees. The RAG question-answering design is a typical customer: it needs filtered top-k search over 46.8 million chunks.
Functional requirements:
Non-functional requirements: a write is visible to queries within 1 second at p99, and a client can ask to read its own writes. For a 10-million-vector, 1,536-dimension index, queries take under 50 ms at p99 (the value 99% of queries beat) at recall@10 of at least 0.95, where recall@10 is the share of the exact 10 nearest neighbours returned (recall, speed and memory). An acknowledged write survives the loss of a node or a zone, and no tenant can push another past its latency target.
Tenants. These figures are invented. Most indexes are tiny and a few are enormous.
| Index size | Indexes | Typical vectors | Total vectors |
|---|---|---|---|
| Tiny, under 100,000 | 54,000 | 10,000 | 0.54 billion |
| Small, 100,000 to 10 million | 5,700 | 1 million | 5.7 billion |
| Large, 10 million to 1 billion | 295 | 40 million | 11.8 billion |
| Huge, over 1 billion | 5 | 2 billion | 10 billion |
| Total | 60,000 | 28.04 billion |
Assume the huge indexes use 768 dimensions, the other 18.04 billion vectors are half 768 and half 1,536, and each vector carries 256 bytes of ID and metadata.
Bytes per vector. int8 scalar quantization stores each number in one byte. Product quantization (PQ) uses one byte per 12 dimensions here. HNSW rows use the FAISS formula from HNSW, d × 4 + M × 2 × 4 bytes, so at M = 16 the links add 128 bytes.
| Form | 768 dims | 1,536 dims |
|---|---|---|
| float32 vector | 3,072 B | 6,144 B |
| int8 vector | 768 B | 1,536 B |
| PQ code | 64 B | 128 B |
| HNSW over float32 | 3,200 B | 6,272 B |
| HNSW over int8 | 896 B | 1,664 B |
Logical size. The huge indexes hold 10 billion × (3,072 + 256) = 33.3 TB. The rest average (3,072 + 6,144) ÷ 2 = 4,608 bytes of float32, so 18.04 billion × 4,864 = 87.7 TB. Customers store 121 TB.
Where the bytes live. Assume allocated costs per GB-month of $3.50 for RAM, $0.10 for local NVMe SSD and $0.023 for object storage, all invented. HNSW over float32 for everything, in RAM with two replicas, needs (10 billion × 3,200 + 18.04 billion × 4,736) × 2 = 234.9 TB, about $822,000 a month. A tiered layout:
| Tier | Holds | Size | Monthly |
|---|---|---|---|
| Object storage | Every segment, the source of truth | 121.0 TB | $2,784 |
| RAM, shared pool | int8 HNSW for the warm 20% of non-huge vectors (3.61 billion × 1,280 B), two replicas | 9.24 TB | $32,328 |
| RAM, huge pools | 64-byte PQ codes for 10 billion vectors, two replicas | 1.28 TB | $4,480 |
| NVMe, shared pool | float32 and metadata of warm vectors, two replicas | 35.1 TB | $3,510 |
| NVMe, huge pools | Vectors, 256-byte graph rows and metadata, two replicas | 71.7 TB | $7,168 |
| Total | about $50,300 |
That is 16 times cheaper, and it rests on an assumption to state aloud: on a given day only about 20% of the non-huge vectors are queried at all. Check it against access logs.
Traffic. Assume 20,000 queries a second on average and 50,000 at peak, 5% of them to the huge indexes. Writes average 10,000 vectors a second (864 million a day, 3.1% of all vectors) and bulk imports reach 60,000. At 4,864 bytes each, the write log carries 48.6 MB/s on average and 292 MB/s at peak.
Nodes. Assume a node with 512 GB of RAM (400 GB usable for index data), 7.5 TB of NVMe and 64 vCPUs.
| Call | Purpose |
|---|---|
POST /v1/indexes | Create an index: dimension, metric, optional sparse support. |
POST /v1/indexes/{ix}/namespaces/{ns}/upsert | Up to 1,000 vectors with IDs and metadata. Returns a write token. |
POST /v1/indexes/{ix}/namespaces/{ns}/delete | Delete by IDs or by filter. |
POST /v1/indexes/{ix}/namespaces/{ns}/query | Top-k with a filter, optional sparse vector, optional write token. |
POST /v1/indexes/{ix}/restore | Restore to a point in time, into a new index. |
{"vector": [0.012, -0.044, ...], "top_k": 10,
"filter": {"lang": {"$eq": "de"}, "product": {"$in": ["billing", "invoices"]}},
"include_metadata": true, "after_write": "wt_7:184022"}
The write token wt_7:184022 means shard 7, log sequence number 184,022. Passing it as after_write asks for read-your-writes.
A query goes to one replica of each shard involved, which searches its cached segments and mutable segment and returns its best results. A node missing a segment fetches it from object storage.
A namespace is stored like an LSM-tree: recent writes in a small mutable segment, older data in immutable segments, files written once and later replaced by merged files.
The mutable segment seals at 5,000 vectors or 10 seconds. A builder reads that log range and writes a segment file with four parts: an HNSW graph over int8 vectors, float32 vectors for rescoring, metadata with an inverted index (each value mapped to the rows that have it), and the vectors grouped by centroid in an IVF layout for reads straight from object storage. It then writes a new manifest, the small file listing the namespace's segments and the highest log sequence number they cover, with a conditional write that fails if another builder got there first.
Each segment is searched separately, so ten 1-million-vector segments cost about ten times one 10-million-vector segment. A tiered merge combines eight segments of similar size, giving tiers of about 5,000, 40,000, 320,000 and 2.5 million vectors, up to a cap of 25 million. From a 5,000-vector flush to the cap takes log₈(25,000,000 ÷ 5,000) ≈ 4.1 merges, so each vector is built about four times. Assuming a core inserts 2,000 vectors a second into HNSW, 10,000 writes a second need 10,000 × 4 ÷ 2,000 = 20 builder cores, and 120 at the import peak, in a pool separate from queries.
A delete writes a tombstone, a log record saying "ID x is deleted as of sequence 184,050", and an upsert of an existing ID is a tombstone plus a new row. Each node keeps a map from ID to the sequence number of its newest write and skips older rows.
Tombstones hurt HNSW. A deleted node stays in the graph, since removing it would cut the paths through it, so the walk passes through but can't return it. With 30% deleted, collecting 10 live results takes about 10 ÷ 0.7 ≈ 14 candidates, and some regions become reachable only through dead nodes. Builders rewrite a segment once 20% of its rows are deleted, a threshold to tune against measured recall, as in compaction and deletes.
A query searches each immutable segment with its index and the mutable segment by brute force, then keeps the best k. Without the newest-sequence check below, the deleted a and the old copy of b would come back.
import heapq
def dot(a, b):
return sum(x * y for x, y in zip(a, b))
def search_segment(query, rows, latest, k):
"""Exact for clarity; a built segment walks its HNSW graph instead."""
live = ((dot(query, vec), vid) for vid, seq, vec in rows
if latest.get(vid) == seq) # skip deleted and overwritten rows
return heapq.nlargest(k, live)
def search_namespace(query, segments, mutable, latest, k):
partial = [search_segment(query, seg, latest, k) for seg in segments + [mutable]]
return heapq.nlargest(k, (hit for hits in partial for hit in hits))
# Rows are (id, sequence number, vector); latest holds each live id's newest write.
built = [[("a", 1, [0.9, 0.1]), ("b", 2, [0.8, 0.3]), ("c", 3, [0.1, 0.9])]]
mutable = [("b", 7, [0.0, 1.0]), ("d", 8, [0.95, 0.05])] # b rewritten, d new
latest = {"b": 7, "c": 3, "d": 8} # a deleted at seq 6
print(search_namespace([1.0, 0.0], built, mutable, latest, k=2))
# [(0.95, 'd'), (0.1, 'c')]
The 5,000-vector cap keeps the scan cheap: 5,000 × 6,144 bytes = 30.7 MB, about 3 ms if a core scans an assumed 10 GB/s. Sealed ranges waiting for a builder are scanned the same way, so a builder backlog shows up as query latency.
Object storage, replicated across zones by its provider, holds every segment, manifest and log range older than a few seconds. Query nodes are caches: int8 graphs in RAM, float32 vectors and metadata on NVMe.
A lost node loses nothing. Its replacement downloads its shards' segments, 400 GB in about 200 seconds at an assumed 2 GB/s, while the other replica serves.
Cold namespaces cost almost nothing. A namespace of 1 million 1,536-dimension vectors is 6.4 GB, $0.15 a month in object storage. Warm, its 1.66 GB of int8 graph on two replicas costs 3.33 GB × $3.50 = $11.65, about 80 times more. Nodes evict namespaces by recency (when the cache is full).
The first query to a cold namespace is slow. An HNSW walk over object storage would need a round trip per hop, so the cold path uses the cluster layout. With an assumed 50 ms to first byte and 1 GB/s:
About 190 ms, which misses the warm target but answers. Meanwhile the node downloads the 1.66 GB graph in about 2 seconds, and later queries are warm. Customers can pay to pin a namespace in memory.
| Segment | Structure | Why |
|---|---|---|
| Mutable, or under 5,000 vectors | Brute force | Under 31 MB to scan, no build, recall 1.0 |
| Up to 25 million vectors | HNSW over int8 in RAM, float32 rescoring from NVMe | High recall, any filter, 41.6 GB at the cap |
| Huge-index shards of 125 million | DiskANN-style graph on NVMe, PQ codes in RAM | 8 GB of RAM per shard instead of 208 GB |
Rescoring corrects compression error: compressed distances pick about 40 candidates, the node reads their float32 vectors, recomputes exact distances and keeps the top 10, the step DiskANN also uses.
Latency budget for a warm query on a 10-million-vector, 1,536-dimension namespace in one shard (p99 figures assumed):
| Step | p99 |
|---|---|
| Client to router, reused connection | 3 ms |
| Auth, shard lookup, admission, hop to the node | 2 ms |
| Evaluate the filter on each segment | 3 ms |
| HNSW over int8, ef_search 128, one large and three small segments | 15 ms |
| Brute force over the mutable segment | 3 ms |
| Rescore 40 candidates from NVMe | 2 ms |
| Merge, read top-10 metadata, reply | 3 ms |
| Total | 31 ms |
The other 19 ms absorbs queueing and pauses. The namespace takes 16.6 GB of RAM and 61.4 GB of NVMe, so it fits one shard, and replicas add throughput.
Hybrid search. With sparse support on, each segment also holds an inverted index of sparse vectors. A hybrid query runs both searches under the same filter and fuses the lists with reciprocal rank fusion (hybrid search).
Filtering describes filtering after the search, before it, or during the graph walk. A shared service can't pick one, because filters range from "every row" to "12 rows". Each segment's inverted index returns a bitmap of matching rows, so the node knows the exact match count before searching and chooses per segment:
def plan(matching: int, total: int, k: int = 10, scan_limit: int = 50_000) -> str:
"""Pick a search strategy for one segment from the filter's exact match count."""
if matching == 0:
return "skip segment"
if matching <= scan_limit:
return f"exact scan of {matching:,} vectors"
selectivity = matching / total
if selectivity >= 0.5:
return f"graph search, then filter {round(k / selectivity * 2)} candidates"
return f"filtered graph walk, ef raised to {min(1000, round(128 / selectivity ** 0.5))}"
for matching in (0, 4_000, 40_000, 9_000_000, 300_000): # in a 10M-vector segment
print(f"{matching / 10_000_000:>7.2%}", plan(matching, 10_000_000))
# 0.00% skip segment
# 0.04% exact scan of 4,000 vectors
# 0.40% exact scan of 40,000 vectors
# 90.00% graph search, then filter 22 candidates
# 3.00% filtered graph walk, ef raised to 739
50,000 int8 vectors at 1,536 dimensions are 76.8 MB, about 8 ms of one core, for an exact answer. In the middle range the walk follows non-matching nodes as stepping stones but never returns them. The square-root rule for ef is a heuristic to tune on measured filtered recall. Qdrant's indexing documentation describes the same approach: payload indexes estimate filter cardinality so the planner can choose a strategy.
A filter every query carries, such as tenant_id, belongs in the namespace instead, since a namespace is its own set of segments. A customer can also declare a partition key, and builders group rows by it inside each segment so a filter on it reads one group.
A shard is a unit of placement with its own log partition and replicas.
Each shard returns its own top k, because the nearest vectors may all sit on one shard:
import heapq
def gather(shard_results, k):
"""Each shard returns its own top k, best first; keep the best k overall."""
return heapq.nlargest(k, (hit for hits in shard_results for hit in hits))
shard_hits = [[(0.97, "p17"), (0.95, "p03"), (0.94, "p88")], # shard 0 holds the best three
[(0.91, "q40"), (0.80, "q02"), (0.79, "q11")],
[(0.89, "r09"), (0.85, "r61"), (0.70, "r30")]]
print(gather(shard_hits, k=3)) # p17, p03, p88
print(gather([h[:1] for h in shard_hits], k=3)) # k / 3 per shard: p17, q40, r09
A query is as slow as its slowest shard. If each shard beats 25 ms 99% of the time, a 16-shard query beats it only 0.99¹⁶ = 85.1% of the time. Holding the query's p99 at 25 ms needs each shard to meet it at the 99.937th percentile (0.99^(1/16)), and with 64 shards 47% of queries would wait on a slow one (the tail at scale). Three measures help:
partial: true.Each shard has two query replicas in different zones, more when hot, and the router picks the one with fewest queries in flight. Writes don't go through replicas. Each log partition has one leader, elected by Raft among three log replicas, so every replica applies the same writes in the same order and "the newest write to ID x" means the same thing everywhere. Writes continue while two of the three log replicas are reachable.
Replicas tail the log independently, so one may lag another by 300 ms, and a customer who upserts and then queries can miss their own write. A query with after_write: "wt_7:184022" goes to a replica of shard 7 that has applied sequence 184,022, or waits up to a second for one, then fails with a retryable error. That is read-your-writes from guarantees for one user. Queries without a token take any replica and may be a second stale, the latency side of PACELC.
A filter matching 60% of a 10-million-vector namespace on an unindexed field would force a full scan of 15.4 GB of int8, about 1.5 seconds of one core, the CPU of 500 ordinary queries. Defences:
Recall falls without errors: deletes pile up, data drifts, a new filter is very selective. For each warm namespace, a job samples up to 1,000 queries a day with their filters, pins the manifest version they ran on, and recomputes the exact top 10 on a separate pool. For 10 million 1,536-dimension vectors, one exact query reads 61.4 GB, 6.1 core-seconds at 10 GB/s, so 1,000 samples cost about 1.7 core-hours a day, less when 50 samples share one pass (exact search).
Recall@10 is reported per namespace and per filter selectivity band, since unfiltered queries hide filtered ones, with an alert when the 7-day figure drops below 0.93. Freshness lag, builder backlog and cold-query rate get their own SLOs.
Vectors from two models can't be compared (changing the embedding model), and dimensions often differ, so a new model means a new index. An alias, a name that points at one index, makes the switch atomic: the customer creates support-docs-v2, writes every change to both indexes, backfills v2, compares them on a test set, repoints the alias, and deletes v1 after a rollback window. They pay for both during the overlap. The RAG design shows these steps from the customer's side.
A backup is a copy of a manifest, since the files it names never change. Restoring to 14:03:20 takes the last manifest before then and replays the log up to that moment, into a new index so a mistake can't overwrite live data.
Garbage collection keeps any segment a retained manifest names. A segment stores about 6,144 bytes per vector (float32, int8, links, metadata), and merging rebuilds each write about four times, so 864 million writes a day produce 864 million × 4 × 6,144 ≈ 21.2 TB of segment files a day, 149 TB over 7 days. That more than doubles object storage bytes but adds only about $3,400 a month, under 7% of the tiers' cost.
Nodes write a usage record per request to a stream, and an aggregator rolls them up per tenant and hour, deduplicating by request ID, as in the inference platform. Warm memory is the hard part to price: the 1-million-vector namespace costs $0.15 cold and $11.65 warm, yet storage bills by logical size, so the storage price must cover the blended cost, and pinning a namespace in memory is sold separately.
Warm, its int8 HNSW is 46.8 million × (1,024 + 128) = 53.9 GB, or 107.8 GB with two replicas, about $377 a month at $3.50 per GB. Cold, its 46.8 million × (4,096 + 256) bytes = 203.7 GB costs about $4.68 in object storage. At 10 queries a second it never goes cold, so $377 is the real cost.
80,000,000 × 1,664 bytes = 133.1 GB. Over the 25-million cap, so 4 shards of 20 million (33.3 GB each), and every query asks all 4. The chance at least one exceeds its own p99 is 1 − 0.99⁴ ≈ 3.9%.
The query reached a replica that hadn't applied that log position yet. Passing the upsert's write token as after_write routes the query to a replica that has, or waits for one.
Deleted nodes remain in the graph as paths but can't be returned, so the walk needs more visits to find k live results and some regions are reachable only through dead nodes. Rewrite the segment without them. The 20% threshold should have done so already, so check the builder backlog.
About 100 × 0.004 = 0.4 results survive on average, so most queries return nothing. The bitmap shows 40,000 matches, under the 50,000 limit, so the node scans them exactly: 61 MB of int8, about 6 ms, recall 1.0.
1 − 0.99⁶⁴ ≈ 47% of queries wait on at least one shard beyond its p99, so the query p99 is set by the shards' 99.98th percentile. Use fewer, larger shards and hedge to a second replica after a shard's p95.
Previous: Design a platform for running AI agents (long-running, tool-using tasks for many customers) · Next: Design an image generation service (text to image, like Midjourney or DALL·E)