System Design Handbook · llms.txt
An employee asks an internal assistant, "What is the limit for a client dinner in Berlin?" The answer should come from the finance wiki, cite the page, reflect yesterday's policy change, and never quote the budget memo the employee isn't allowed to open. If no document answers the question, the assistant should say so.
This is retrieval-augmented generation (RAG) at company scale: four sources, tens of millions of chunks, permissions that differ per user, and documents that change every minute. Retrieval-augmented generation explains the technique and embeddings and vector search the indexes. This chapter makes the decisions and puts numbers on them. Streaming and conversation storage work as in the chat assistant design.
Functional requirements:
Non-functional requirements: an edit is answerable within 5 minutes at the 95th percentile (p95, the value 95% of cases beat), a revoked permission takes effect within 2 minutes, and the first token arrives within 2 seconds at p95. Actions such as filing a ticket are out of scope.
These invented assumptions describe the company, called Larkfield:
Corpus. A chunk is 400 tokens and starts 50 tokens before the previous one ends, so each new chunk adds 350 tokens, close to the RAG chapter's 300-word chunks with 50-word overlap. A document of T tokens needs (T − 50) ÷ 350 chunks, rounded up: a 910-token wiki page needs 2.46, so 3, and a 2,340-token drive file needs 6.54, so 7. Chat messages are grouped by thread into 400-token chunks.
| Source | Documents | Words each | Tokens each | Chunks each | Chunks |
|---|---|---|---|---|---|
| Wiki pages | 600,000 | 700 | 910 | 3 | 1,800,000 |
| Tickets | 8,000,000 | 250 | 325 | 1 | 8,000,000 |
| Drive files | 2,500,000 | 1,800 | 2,340 | 7 | 17,500,000 |
| Chat messages | 400,000,000 | 15 | 19.5 | grouped | 19,500,000 |
| Total | 46,800,000 |
The words total 12.92 billion, about 16.8 billion tokens, and chat's 7.8 billion tokens ÷ 400 give 19.5 million chunks. With overlap, the embedding model reads about 17.6 billion tokens.
Index memory. The embedding model returns 1,024 numbers per chunk, as in the RAG chapter. HNSW rows use the FAISS formula from HNSW, and int8 (scalar quantization) stores each number in one byte, losing a little precision.
| Storage | Bytes per chunk | 46.8 million chunks |
|---|---|---|
| float32 vectors | 1,024 × 4 = 4,096 | 191.7 GB |
| HNSW over float32, M = 16 | 4,096 + 16 × 2 × 4 = 4,224 | 197.7 GB |
| HNSW over int8 | 1,024 + 128 = 1,152 | 53.9 GB |
| PQ, 64-byte codes | 64 | 3.0 GB |
Embedding cost. At an assumed $0.02 per million tokens, the backfill's 17.6 billion tokens cost 17,600 × $0.02 = $352. At the vector search chapter's assumed 1,000 chunks a second per GPU, it takes 46,800 GPU-seconds, about 13 GPU-hours. The sources are slower: at 50 downloads a second, 2.5 million drive files take 50,000 seconds, about 14 hours.
Daily changes. Assume 4,000 wiki edits (3 chunks each), 50,000 new or updated tickets, 20,000 changed drive files (7 chunks each) and 1.2 million chat messages. That is 12,000 + 50,000 + 140,000 + 58,500 = 260,500 chunks a day (chat: 1.2 million × 19.5 ÷ 400), or 0.56% of the index. Embedding 260,500 × 400 ≈ 104 million tokens costs about $2.08, and the rate, 8 chunks a second over 9 hours, is trivial for one GPU.
Questions. 20,000 users × 6 = 120,000 a day, or 120,000 ÷ 32,400 s ≈ 3.7 a second, and 9.3 at peak. The design plans for 10 a second. Over 22 working days that is 2.64 million questions a month, and 40% are follow-ups.
Tokens per question, using the token budget layout:
| Part | First question | Follow-up |
|---|---|---|
| Instructions | 500 | 500 |
| Recent turns | 0 | 600 |
| 8 chunks × (400 + 25 for title and source line) | 3,400 | 3,400 |
| Question | 40 | 40 |
| Input | 3,940 | 4,540 |
| Answer (average, capped at 600) | 300 | 300 |
The average input is 0.6 × 3,940 + 0.4 × 4,540 = 4,180 tokens.
LLM cost. With the chat assistant design's assumed prices of $3 per million input tokens and $15 per million output tokens, a question costs 4,180 × $3 ÷ 10⁶ = $0.01254 plus 300 × $15 ÷ 10⁶ = $0.0045. Rewriting follow-ups on a small model at an assumed $0.25 and $1.25 per million (700 tokens in, 40 out) adds 0.4 × $0.000225 = $0.00009. That is about $0.0171 a question, or 2.64 million × $0.01713 ≈ $45,200 a month. At peak the model reads 41,800 input tokens a second, which the provider's rate limits must allow.
Self-hosting check. On the handbook's 70B replica (four H100s), prefill costs 70.8 µs per token at 50% of peak, so 10 questions a second need 10 × 4,180 × 70.8 µs ≈ 2.96 replica-seconds of prefill each second. The coupled model from estimating AI workloads then needs 5 replicas, plus a spare: 24 H100s, $52,560 a month at $3 an hour, idle at night. So the design calls a hosted model through an LLM gateway.
Asking returns a server-sent event (SSE) stream, as in the chat assistant, with sources before the answer.
| Call | Purpose |
|---|---|
POST /v1/ask | Ask, optionally in a conversation_id. Streams sources, the answer, then verified citations. |
POST /v1/answers/{id}/feedback | Thumbs up or down, a reason, and the citations clicked. |
POST /v1/connectors, GET /v1/connectors/{id} | Admin: add a source; read its sync state, lag and parse errors. |
POST /v1/connectors/{id}/resync | Admin: list the source again and repair differences. |
POST /hooks/{source} | Receives change notifications from a source. |
event: sources
data: {"answer_id": "a_71c2", "sources": [{"n": 1, "title": "Travel and expenses",
"section": "Meals > Client dinners", "url": "...", "updated": "2026-10-24"}]}
event: delta
data: {"text": "Client dinners in Berlin are capped at EUR 90 per person [1]."}
event: citations
data: {"verified": [1], "removed": [], "abstained": false}For one question:
A connector turns a source's changes into events of (document ID, version, change type), using the best mechanism the source offers:
A nightly reconciliation also compares every document ID and version in the source with the index and queues the differences, which catches lost webhooks and, above all, lost deletes.
Events go to a log partitioned by document ID, so one worker handles a document's changes in order (choosing a partition key) and skips versions older than the indexed one. Backfills use a low-priority topic, so they can't delay a wiki edit. Of the 5-minute budget, the event takes up to 30 seconds to arrive (2 minutes when polled), fetching and parsing a few seconds, and embedding and the index refresh about a second each. Alert when the oldest waiting event is 3 minutes old.
PDFs need a layout-aware parser that reads columns in order, and scanned ones go to a slower OCR (optical character recognition) queue. Table chunks repeat the header row, since "Berlin | 90 | EUR" means nothing without "City | Limit | Currency". Each slide is a chunk with the deck title and its notes, and a ticket is one chunk of title, description, resolution and recent comments. Chat is grouped by thread, or by channel into windows that close at 400 tokens or a 10-minute gap, each starting with the channel name and date. Parsers fail quietly (the indexing pipeline), so flag files that yield almost no text.
Documents are split by section, following chunking, and a section over 400 tokens is split into parts with 50 tokens of overlap. A chunk's ID is the document ID, heading path and part number, and it stores a hash of its text. The heading path is embedded with the text, so a chunk under "VPN > Login" still says "VPN".
Section IDs make edits cheap. A fixed 400-token window shifts every later boundary when a word is inserted, so a one-line edit changes every later chunk. With section IDs, an edit to one section changes one chunk's hash, and only that chunk is embedded again. Contextual retrieval would add 46.8 million model calls to the backfill, so try it on the wiki and drives first.
The same PDF is often attached to 30 tickets and copied into 3 folders. An embedding cache keyed by text hash embeds it once, but each copy keeps its own index entry, permissions and link, since a user may read one copy and not another. At query time, identical chunks collapse to one readable copy, so they don't fill the 8 slots.
Each chunk carries metadata: document ID and version, title, heading path, URL, date, text hash, embedding model, and the ACL principals, the user, group and channel IDs allowed to read it.
"And for Paris?" retrieves nothing alone, so a small model reads the last two turns and returns "client dinner spending limit in Paris" in about 230 ms (query rewriting).
Larkfield needs both searches. "How do I fix ERR-4012?" needs the exact error code, which BM25 finds and vectors often miss. "Can I expense a laptop stand?" should match "home office equipment", which vectors find and BM25 misses. Both run on every shard with the permission filter inside, return 50 candidates each, and reciprocal rank fusion (RRF) merges them by rank (hybrid search).
The sketch shows why the filter belongs inside the search. A new hire can read only the "all-staff" documents, 2% of the corpus. Post-filtering the top 100 vector results leaves 2 chunks. Filtering inside each search keeps going until 50 readable chunks are found, and RRF merges the two lists into 84 distinct candidates.
import random
def allowed(chunk, principals):
return not chunk["acl"].isdisjoint(principals) # any shared principal grants access
def filtered_search(ranked, principals, k):
out = []
for c in ranked: # the engine walks until k pass
if allowed(c, principals):
out.append(c)
if len(out) == k:
break
return out
def rrf(*rankings, k=60):
scores = {}
for ranking in rankings:
for rank, c in enumerate(ranking, start=1):
scores[c["id"]] = scores.get(c["id"], 0.0) + 1 / (k + rank)
return sorted(scores.items(), key=lambda item: item[1], reverse=True)
random.seed(4)
# Every 50th chunk is readable by all staff; the rest belong to 37 teams.
chunks = [{"id": i, "acl": {"all-staff"} if i % 50 == 0 else {f"team-{i % 37}"}}
for i in range(10_000)]
bm25, vector = random.sample(chunks, 10_000), random.sample(chunks, 10_000)
new_hire = {"user-4410", "all-staff"}
print(sum(allowed(c, new_hire) for c in vector[:100])) # post-filter: 2
fused = rrf(filtered_search(bm25, new_hire, 50), filtered_search(vector, new_hire, 50))
print(len(fused), all(i % 50 == 0 for i, _ in fused)) # 84 True
A cross-encoder scores the query and each candidate read together (reranking with a cross-encoder). Fifty candidates of 400 tokens plus 30 of query make 50 × 430 = 21,500 tokens per question. For an assumed 0.5-billion-parameter cross-encoder at 2 FLOPs per parameter per token, that is 2 × 0.5 × 10⁹ × 21,500 = 21.5 TFLOP. One H100 at 50% of its 989 dense BF16 TFLOPS does it in 21.5 ÷ 494.5 ≈ 43 ms. At 10 questions a second one GPU is 43% busy, so the design runs two for short queues and failures. Reranking 150 would triple the cost, so measure recall@8 at 50 and 100 first.
The 8 chunks enter the prompt numbered, best first, each with title, section path and date (order and citations). The instructions say to answer only from the sources, cite each claim as [n], prefer the newest source when two disagree, and say so when the sources don't answer. When the top rerank score is below a threshold, the service abstains without calling the model. Set the threshold on the golden set, for example so that 90% of unanswerable questions abstain and under 5% of answerable ones do.
After the answer streams, a verifier removes any citation number that wasn't sent. A small entailment model, which judges whether one text supports another, checks each cited chunk against its sentence, and unsupported citations are removed and the sentence marked unverified. This runs after the last token, so it doesn't delay the first.
| Step | Time |
|---|---|
| Auth, principals from cache | 15 ms |
| Query rewrite (follow-ups only) | 230 ms |
| Query embedding | 15 ms |
| BM25 and kNN on all shards, in parallel | 60 ms |
| RRF and duplicate collapse | 5 ms |
| Rerank 50 candidates | 50 ms |
| ACL recheck, chunk text, prompt | 20 ms |
| Hosted model queue and prefill, about 4,200 tokens | 700 ms |
| Total for a follow-up | 1,095 ms |
A first question skips the rewrite and takes 865 ms. The 700 ms is an assumed provider figure. A self-hosted 70B replica would spend 4,180 × 70.8 µs ≈ 296 ms on prefill alone. The sources event leaves at about 400 ms.
The document side, which principals may read a document, comes from each source (space permissions, project roles, sharing, channel membership) and is copied onto every chunk. The user side comes from the directory, with nested groups flattened, so a member of "Payroll", inside "Finance", holds both. A typical employee has about 150 principals (one user ID, 90 groups, 60 channels), and each search keeps chunks whose principals intersect those 150, a fast term filter.
Two policies keep the sync safe. A document whose ACL can't be read gets no principals until the next sync (fail closed). A file shared as "anyone with the link" is indexed only for its owner and named principals, or it would become searchable company-wide.
Permissions at retrieval time explains the leak risk, and post-filtering also breaks retrieval. A user who can read 10% of chunks keeps about 10 of the top 100 candidates, and the new hire about 2. Getting 50 readable candidates at 2% would take 50 ÷ 0.02 = 2,500 per search. The restricted documents are often the relevant ones, too, such as finance documents for a finance analyst. Every search path uses one query builder that adds the filter, and a test runs a no-access persona through each path, as in how RAG fails.
When someone leaves the "Finance" group, the directory's event makes the answer service drop that user's cached principals, and a 60-second time-to-live bounds the delay if the event is lost. The index doesn't change.
A sharing change on a folder is harder, since every chunk of every file in it needs new principals, a metadata update with no new embedding. A folder of 50,000 files has 350,000 chunks, and at an assumed 5,000 updates a second the index catches up in 70 seconds. The ACL store updates within seconds, so the recheck of the 8 chunks' documents (step 4 above) covers the gap.
HNSW over int8 takes 53.9 GB across shards, gives high recall, and accepts inserts one at a time, which suits 260,500 changes a day. IVF-PQ (product quantization) fits in 3.0 GB, but its clusters and codebooks are trained on a sample and drift as content changes, and it loses recall unless candidates are rescored.
The design runs HNSW over int8 in a search engine that also holds the BM25 index, so one query builder filters both. Each shard rescores its top 50 with float32 vectors from SSD, reading 50 × 4 KB = 200 KB per query.
Chunks are spread across 6 shards by a hash of the document ID, so a delete touches one shard. Each shard holds 7.8 million chunks: 9.0 GB of int8 graph in memory, about 12 GB of text (17.6 billion tokens × 4 characters ÷ 6) and 32 GB of float32 vectors on SSD. Two replicas per shard make 12 nodes of 64 GB, there for failures, since 10 questions a second is light load.
Sharding by source would let one source be rebuilt alone, but chat holds 42% of the chunks and the wiki under 4%, and every question searches every source. A product sold to many companies would give each customer its own shards, so a filter bug couldn't cross between them.
For a user who can read 2%, most nodes the HNSW walk visits fail the filter, so it must visit about 50 times as many, and recall can drop. Scanning the matching set exactly can be faster: 2% of a shard is 156,000 vectors, 160 MB of int8, and engines such as Lucene switch to an exact scan when a filter is selective enough. Test recall with low-access personas, since an all-access test account hides this (filtering).
Updates. A new version writes changed chunks, deletes removed ones, then becomes current. The prompt builder drops chunks older than their document's current version, so a reader never sees a mix (keeping the index fresh).
Deletes. A delete writes a tombstone, a record that says "document X is deleted as of version 12". It is applied to the index, the chunk store and the answer cache, and kept for 30 days, so a late update with an older version can't revive the document. A check then searches for a phrase from it and expects nothing.
A new embedding model means a new index, since vectors from two models can't be compared (changing the embedding model). Build it blue/green:
Dual writes start before the backfill, so a change made during it reaches green either way. The graphs need about 108 GB during the switch.
The golden set (building an eval set) has 400 real questions with labeled documents, sections and reference answers, 100 the corpus can't answer, and 100 permission probes asked as personas who must not see the answer. Labels name sections, since chunk IDs change with the chunker.
| Metric | What it checks | Target |
|---|---|---|
| Recall@50 before reranking | The first stage finds the evidence | 0.95 |
| Recall@8 after reranking | The evidence reaches the prompt | 0.85 |
| Faithfulness and citation precision | Claims and each [n] are supported | 0.95 |
| Correct abstention | Unanswerable questions get "I don't know" | 0.90 |
| Permission probes | Restricted text appears in an answer | 0, release blocker |
Retrieval metrics defines recall@k. Report each metric per source, because an average can hide a broken connector.
A model grades faithfulness and correctness. Zheng and colleagues found that model judges favor the first of two answers, favor longer answers, and can favor their own outputs, so calibrate the judge against about 200 human labels and use a different model family from the answerer (model-graded evaluation). Online, track thumbs-down, citation clicks, abstentions and rephrased repeats, and grade a 1% sample (sampled judging).
Prompt injection. Any employee can edit the wiki and customers write ticket text, so a planted instruction is retrieved like any chunk (prompt injection). The assistant has no tools, so it can't be steered into acting. Sources are delimited and labeled by type, and the interface renders no images from answers and links only to allowlisted company domains, since a URL can carry data out (images and links are a data channel).
Leaks between users come from an unfiltered search path, a cache without the fingerprint, or logs holding answer text, so logs get access controls. Never fine-tune on the corpus, since weights can't enforce per-user permissions.
A month at peak sizing:
| Item | Monthly cost |
|---|---|
| Hosted LLM, 2.64 million questions × $0.0171 | $45,200 |
| Two reranker H100s, 2 × $3 × 730 hours | $4,380 |
| Search cluster, 12 nodes at an assumed $500 | $6,000 |
| Embedding changes, 22 days × $2.08 | about $46 |
| Total | about $55,600 |
The model is 81% of the bill. Five chunks instead of eight saves 3 × 425 = 1,275 input tokens, $0.0038 a question or about $10,100 a month, if recall@5 holds. Abstaining early on an assumed 8% of questions saves about $3,600, the prefix cache about $3,560, and a 10% answer-cache hit rate about $4,500.
The extra files add 2.5 million × 7 = 17.5 million chunks, for 64.3 million in all. At 1,152 bytes each, the graphs need 64,300,000 × 1,152 ≈ 74.1 GB, about 12.3 GB per shard, which still fits 64 GB nodes. The float32 copies on SSD grow to about 263 GB.
About 100 × 0.02 = 2 chunks survive, so the assistant says "I don't know" to answerable questions. Put the principal filter inside both searches, and let the engine scan a small matching set exactly.
Only the answer service's cached principals, until the directory event invalidates them, or at most 60 seconds through the cache's time-to-live. The index needs no change, since the documents' principals haven't changed. Once the principals refresh, the permission fingerprint changes too, so the answer cache can't serve them a Finance answer. The worst case is about 60 seconds, inside the 2-minute target.
The model uses what it gets, so the evidence isn't arriving. Break recall@50 and recall@8 down by source, since the average can hide a low chat number. Then check windows that split discussions, a lagging chat connector, or channel memberships missing from users' principals.
The delimiters and the instruction to treat sources as data failed, as no prompt reliably stops injection. Having no tools and linking only to allowlisted domains kept the damage small. Find the chunk, mark customer-written fields as untrusted, and add the case to the golden set.
Previous: Design an LLM gateway (one API in front of many model providers for a whole company) · Next: Design an AI coding assistant (inline completions and an agent mode in the editor)