System Design Handbook · llms.txt
Inference is the work a model server does for each request. It takes the prompt, runs a trained model, and returns the reply. Training happens once, in a separate and much larger job. Inference runs for every message a user sends, so its speed and cost decide what a product can offer.
A large language model (LLM) predicts the next token of a sequence. A token is a short piece of text, often a word or part of one. Hugging Face's guide to chat templates puts the idea this way: "All causal LMs, whether chat-trained or not, continue a sequence of tokens." An answer therefore comes out one token at a time, and each token depends on every token before it.
This chapter defines the vocabulary the rest of the AI part uses: prefill and decode, the KV cache (the keys and values an attention step keeps for every token so far), time to first token (TTFT, the wait before the first answer token) and time per output token (TPOT, the average gap between later tokens). Each one gets a concrete example before the general rule. The running example is Meta's Llama 3.1 family, with 8B, 70B and 405B parameters (the learned numbers inside the model). The Llama 3 paper publishes the shape of each size, and the memory arithmetic depends on those shapes.
For GPU hardware, serving engines and fleet sizing, see GPUs and accelerators, inference engines and estimating AI workloads.
A tokenizer splits text into pieces called tokens and maps each piece to an ID in a fixed vocabulary. The model reads and writes IDs, never letters, and a detokenizer turns IDs back into text.
The OpenAI Cookbook's token-counting notebook shows how tiktoken, OpenAI's open-source tokenizer library, splits the sentence "tiktoken is great!" under its o200k_base encoding:
Token ID Piece
1 83 "t"
2 8251 "ikt"
3 2488 "oken"
4 382 " is"
5 2212 " great"
6 0 "!"
Six tokens cover three words and a punctuation mark. Each space attaches to the token after it (" is", " great"). Even "tiktoken" splits into "t", "ikt" and "oken", so token counts and word counts differ. Under cl100k_base, the single word "antidisestablishmentarianism" becomes six tokens.
The merge rules come from byte pair encoding (BPE). tiktoken's README calls BPE a way of converting text into tokens, and its educational module shows the procedure: count adjacent pairs, merge the most frequent pair into a new token, and repeat. Sennrich and colleagues used the same idea to split rare words for machine translation. The script below runs the procedure on a tiny corpus, starting from characters rather than bytes.
from collections import Counter
def apply_merge(pieces, pair):
out, i = [], 0
while i < len(pieces):
if i + 1 < len(pieces) and (pieces[i], pieces[i + 1]) == pair:
out.append(pieces[i] + pieces[i + 1])
i += 2
else:
out.append(pieces[i])
i += 1
return out
def learn_merges(text, num_merges): # start from characters, merge the most common pair
words, merges = Counter(tuple(w) for w in text.split()), [] # word -> count
for _ in range(num_merges):
pairs = Counter()
for word, freq in words.items():
for pair in zip(word, word[1:]):
pairs[pair] += freq
if not pairs:
break
best = max(pairs, key=pairs.get) # ties go to the pair seen first
merges.append(best)
new_words = Counter()
for word, freq in words.items():
new_words[tuple(apply_merge(list(word), best))] += freq
words = new_words
return merges
merges = learn_merges("low low low lower lower newest newest widest", 4)
print("merges:", ["".join(p) for p in merges])
# merges: ['lo', 'low', 'es', 'est']
for word in ["lowest", "widest"]: # encode: apply the merges in learned order
pieces = list(word)
for pair in merges:
pieces = apply_merge(pieces, pair)
print(word, "->", pieces)
# lowest -> ['low', 'est']
# widest -> ['w', 'i', 'd', 'est']
The first merge is "lo", because the pairs l-o and o-w tie at five and the first one counted wins. Encoding applies the merges in the order they were learned, so "lowest" becomes "low" and "est". Frequent pieces become single tokens, and rare text falls back to smaller pieces, so any input can be encoded.
Llama 3's vocabulary has 128K tokens: 100K from tiktoken plus 28K added for better non-English coverage, according to its paper. On every pass the model produces one score per vocabulary entry. A softmax, which turns scores into probabilities that sum to one, converts them into next-token probabilities.
Tokens matter here for four reasons. Context windows count tokens. Providers price input and output by the token, and Anthropic's documentation lists those prices for each model. Decode makes one pass per output token, so output length sets the work. And each token adds a fixed slice to the KV cache.
A chat request is a list of role-tagged messages. The server renders that list into one token sequence. A chat template adds control tokens that mark where each message starts and ends. Hugging Face's guide says the messages become "a token sequence, often with control tokens," which let the model see the chat structure.
The model then runs in three stages.
Llama 3 uses what its paper calls "a standard, dense Transformer architecture," and Meta describes it as a decoder-only model. Decoder-only means each position attends only to earlier tokens, which the Transformer paper enforces with a mask on the decoder's self-attention.
The 8B model has 32 layers, the 70B has 80 and the 405B has 126. Each layer has an attention step, where a token gathers information from the tokens before it, and a feed-forward network, which transforms each token's vector on its own. In "The trophy did not fit in the suitcase because it was too big," the word "it" refers back to "trophy," and attention is the step that lets the model make that link. Attention also drives the cost of long prompts: each token compares itself with every earlier token. The Transformer paper's table gives self-attention cost per layer as O(n²·d), where n is the sequence length, so doubling the prompt roughly quadruples that part of the work.
Generation ends when the model emits its end-of-sequence token, a special token that marks the end of a reply. It also ends at the output limit (max_new_tokens in Hugging Face, which counts "ignoring the number of tokens in the prompt"), at a custom stop string, or when the context window fills. Anthropic's API reports which one happened. end_turn means Claude "finished its response naturally," and max_tokens means it "reached the max_tokens limit specified in your request."
Take a 2,000-token prompt. The server can read all 2,000 tokens in one pass, but writing a 300-token reply takes 300 passes. Prefill is the first pass, over the whole prompt at once. The prompt is known in advance, so the GPU can process all of its tokens together as large matrix operations. Splitwise, a paper about running the two phases on separate machines, calls the prompt phase "a compute-intensive prompt computation." Decode produces the answer one token per pass. Each pass processes only the newest token, but it must read all the weights and the cached keys and values from GPU memory. Splitwise calls this "a memory-intensive token generation." Pope and colleagues put it in hardware terms: each step loads the parameters and KV cache from high-bandwidth memory (HBM) into the compute cores, and at small batch sizes "the time to load weights dominates."
The running example puts numbers on this. It uses a 70B model with BF16 weights (16-bit numbers, 2 bytes each) split across four NVIDIA H100 GPUs with tensor parallelism, which divides each layer's weights among the GPUs. Hugging Face's optimization guide gives the rule of thumb of about 2 GB per billion parameters in bfloat16, so the weights take 140 GB. A FLOP is one floating-point operation. The EleutherAI guide gives the cost of a forward pass as about 2PD FLOPs, where P is the parameter count and D the token count. So one token costs about 2 × 70 × 10^9 = 140 GFLOP.
Prefill reads each weight once and applies it to every prompt token, so it does a lot of arithmetic per byte read. Decode reads every weight to produce one token. The GPU chapter explains this with the roofline model, which compares a chip's arithmetic speed with its memory speed. A longer prompt takes longer to prefill, so long prompts raise TTFT.
Consider one 4,096-token conversation on the 70B model. Its cache takes about 1.34 GB (4,096 × 327,680 bytes, computed below), next to 140 GB of weights.
Each attention step needs the keys and values of every earlier token in every layer. Without a cache, each decode step would recompute them for the whole sequence, "even though the earlier tokens haven't changed," as Hugging Face's KV cache guide puts it. With a cache, each token's keys and values are computed once, stored, and appended to. Hugging Face calls this a trade-off between speed and memory.
The cache size per token follows from the model's shape:
bytes per token = 2 × layers × KV heads × head dimension × bytes per value
The leading 2 covers keys and values. The head dimension is the model dimension divided by the number of attention heads, so the 70B model has 8,192 ÷ 64 = 128. Llama 3 uses grouped-query attention (GQA) with 8 key-value heads at every size, so several query heads share each key-value head. The GQA paper describes the key-value heads as "more than one, less than number of query heads." The vLLM paper runs a similar calculation for OPT-13B, a 13-billion-parameter model, and reports 800 KB per token in FP16, which matches 2 × 5,120 × 40 × 2 = 819,200 bytes.
| Model | Layers | KV heads | Head dimension | Bytes per token, BF16 |
|---|---|---|---|---|
| Llama 3.1 8B | 32 | 8 | 128 | 2 × 32 × 8 × 128 × 2 = 131,072 |
| Llama 3.1 70B | 80 | 8 | 128 | 2 × 80 × 8 × 128 × 2 = 327,680 |
| Llama 3.1 405B | 126 | 8 | 128 | 2 × 126 × 8 × 128 × 2 = 516,096 |
# Bytes of KV cache per token: keys and values (the 2), for every layer and KV head.
LLAMA_3_1 = {
"8B": dict(layers=32, kv_heads=8, head_dim=128),
"70B": dict(layers=80, kv_heads=8, head_dim=128),
"405B": dict(layers=126, kv_heads=8, head_dim=128),
}
def kv_bytes_per_token(layers: int, kv_heads: int, head_dim: int,
bytes_per_value: int = 2) -> int:
return 2 * layers * kv_heads * head_dim * bytes_per_value
for name, shape in LLAMA_3_1.items():
print(f"{name}: {kv_bytes_per_token(**shape):,} bytes per token")
# 8B: 131,072 bytes per token
# 70B: 327,680 bytes per token
# 405B: 516,096 bytes per token
per_token = kv_bytes_per_token(**LLAMA_3_1["70B"])
print(f"70B, one 4,096-token conversation: {4096 * per_token / 1e9:.2f} GB")
# 70B, one 4,096-token conversation: 1.34 GB
cache_budget = 148 * 10**9 # bytes left after the weights, in the running example
print(f"70B cache budget holds {cache_budget // per_token:,} tokens")
# 70B cache budget holds 451,660 tokens
print(f"Without GQA (64 KV heads): {kv_bytes_per_token(80, 64, 128):,} bytes per token")
# Without GQA (64 KV heads): 2,621,440 bytes per token
One 4,096-token conversation takes 4,096 × 327,680 = 1,342,177,280 bytes, about 1.34 GB. Without GQA, the 70B model would keep 64 key-value heads, and the same formula gives 2,621,440 bytes per token, eight times as much.
In the running example, four 80 GB H100s hold 320 GB. Keeping an assumed 10% (32 GB) free for activations and runtime, and loading 140 GB of weights, leaves 148 GB for the cache. That holds 148 × 10^9 ÷ 327,680 ≈ 451,660 tokens, about 110 conversations of 4,096 tokens (451,660 ÷ 4,096 ≈ 110).
The cache grows with every token, and each request has its own. The vLLM paper observes that it "grows and shrinks dynamically" and that memory "can be significantly wasted by fragmentation and redundant duplication." The managing KV cache memory and prefix caching sections of the inference engines chapter cover how engines cope.
Suppose the model has just read "The cat sat on the" and gives "mat" a score 2.0 higher than each of seven other entries. Sampling is the rule that picks the next token from those scores. Four rules matter.
scores / self.temperature. Values below 1 sharpen the distribution, and 0 is "equivalent to shifting all probability mass to the most likely token," which is greedy decoding. Temperature only matters when sampling is on.The script below uses a fake model that scores the right next token 2.0 and every other token 0.0. It shows greedy decoding, sampling at temperature 1, and both stop conditions.
import math
import random
VOCAB = ["The", "cat", "sat", "on", "the", "mat", ".", "<eos>"]
NEXT = {"The": "cat", "cat": "sat", "sat": "on", "on": "the", "the": "mat", "mat": ".", ".": "<eos>"}
def forward(tokens, kv_cache): # fake model: caches each token, scores the next one
kv_cache.extend(tokens) # a real engine stores keys and values per layer
scores = [0.0] * len(VOCAB)
scores[VOCAB.index(NEXT[tokens[-1]])] = 2.0 # the right next token scores highest
return scores
def sample(scores, temperature, rng):
if temperature == 0:
return VOCAB[scores.index(max(scores))] # greedy
weights = [math.exp(s / temperature) for s in scores]
return rng.choices(VOCAB, weights=weights)[0]
def generate(prompt, max_new_tokens, temperature):
rng, kv_cache, output = random.Random(0), [], []
scores = forward(prompt, kv_cache) # prefill: the whole prompt in one pass
while len(output) < max_new_tokens: # decode: one token per pass
token = sample(scores, temperature, rng)
output.append(token)
if token == "<eos>": # stop condition 1: end-of-sequence token
return output, "end-of-sequence", len(kv_cache)
scores = forward([token], kv_cache)
return output, "max_new_tokens", len(kv_cache) # stop condition 2: the limit
for limit, temp in [(10, 0), (3, 0), (10, 1.0)]:
output, reason, cache = generate(["The"], limit, temp)
print(" ".join(output), "|", reason, "| cache:", cache)
# cat sat on the mat . <eos> | end-of-sequence | cache: 7
# cat sat on | max_new_tokens | cache: 4
# mat . . on the mat . the mat . | max_new_tokens | cache: 11
for t in (0.5, 1.0, 2.0):
p = math.exp(2 / t) / (math.exp(2 / t) + len(VOCAB) - 1)
print(f"temperature {t}: right token {p:.2f}")
# temperature 0.5: right token 0.89
# temperature 1.0: right token 0.51
# temperature 2.0: right token 0.28
Greedy decoding writes the whole sentence and stops at the end-of-sequence token, with seven cache entries, one for each token read. A limit of three stops on the limit with four entries, because the newest token has not run through the model yet. At temperature 1 the fake model wanders, and the right token's probability is 0.51. It is 0.89 at temperature 0.5 and 0.28 at 2.0.
For extraction or classification, where one answer is right, use greedy decoding or a low temperature. For brainstorming, use a higher temperature with top-p.
Here is one streamed reply from Anthropic's API, abridged with the ping events removed. Most events carry one piece of the answer.
event: message_start
data: {"type": "message_start", "message": {"id": "msg_...", "role": "assistant", "content": [], ...}}
event: content_block_start
data: {"type": "content_block_start", "index": 0, "content_block": {"type": "text", "text": ""}}
event: content_block_delta
data: {"type": "content_block_delta", "index": 0, "delta": {"type": "text_delta", "text": "Hello"}}
event: content_block_delta
data: {"type": "content_block_delta", "index": 0, "delta": {"type": "text_delta", "text": "!"}}
event: content_block_stop
data: {"type": "content_block_stop", "index": 0}
event: message_delta
data: {"type": "message_delta", "delta": {"stop_reason": "end_turn", "stop_sequence": null}, ...}
event: message_stop
data: {"type": "message_stop"}
Streaming sends each token as soon as it is decoded, usually as server-sent events (SSE). Anthropic's API uses SSE when a request sets "stream": true. MDN describes the format: the content type is text/event-stream, a blank line ends each event, and lines starting with data: carry its payload. The connection runs one way, from server to client.
import json
def sse_payloads(lines): # yields the data field of each event; a blank line ends an event
data = []
for line in lines:
line = line.rstrip("\n")
if line == "":
if data:
yield "\n".join(data)
data = []
elif line.startswith("data:"): # event: and comment lines are ignored
value = line[5:]
data.append(value[1:] if value.startswith(" ") else value)
def fake_server(): # stand-in for an HTTP body: one event per token, then a stop event
for piece in ["Hello", " there", "!"]:
event = {"type": "content_block_delta", "delta": {"type": "text_delta", "text": piece}}
yield "event: content_block_delta"
yield "data: " + json.dumps(event)
yield ""
yield "event: message_stop"
yield 'data: {"type": "message_stop"}'
yield ""
text = ""
for payload in sse_payloads(fake_server()):
event = json.loads(payload)
if event["type"] == "content_block_delta":
text += event["delta"]["text"]
print(f"token: {event['delta']['text']!r}")
elif event["type"] == "message_stop":
print("stream finished:", repr(text))
# token: 'Hello'
# token: ' there'
# token: '!'
# stream finished: 'Hello there!'
Streaming changes when the first token appears. The full answer still takes as long as before, because every decode step still runs. NVIDIA's documentation says time to first token "measures how long you wait before seeing the model's output."
Each hop must forward events as they arrive. A gateway or proxy that buffers the response hides the stream, and the user waits for the whole reply.
When the client disconnects, the server should stop decoding, or the GPU keeps producing tokens nobody reads. The realtime and protocols and APIs chapters cover SSE and other transports.
Picture one response. The request waits in a queue, prefill runs, the first token appears, and then the rest arrive one at a time.
Two numbers describe that timeline. NVIDIA's benchmarking documentation, as of October 2026, defines them and two related measures:
Other tools define these differently, and NVIDIA notes that results should be compared "only when definitions align." With these definitions, end-to-end latency ≈ TTFT + (output tokens − 1) × TPOT. Assume a TTFT of 200 ms and a TPOT of 13.6 ms, the running example's step time below. A 300-token answer takes 200 + 299 × 13.6 = 4,266.4 ms, about 4.3 seconds.
| Situation | Time per step | Tokens a second, total | Per user |
|---|---|---|---|
| One user | 140 GB ÷ 13.4 TB/s ≈ 10.4 ms | about 96 | about 96 |
| 64 users, 2,000 tokens each | (140 + 42) GB ÷ 13.4 TB/s ≈ 13.6 ms | about 4,700 | about 74 |
The second row's cache reads come to 64 × 2,000 × 327,680 bytes ≈ 42 GB. Each step makes one token per user, so the server makes 64 ÷ 0.0136 ≈ 4,706 tokens a second, or about 74 for each user. The step's arithmetic, 64 × 140 GFLOP ≈ 9 TFLOP, takes about 2.3 ms at full speed, so the step still waits on memory.
Batching raises total throughput about 49 times (4,706 ÷ 96 ≈ 49) and cuts each user's speed by about 23%. The weights are read once per step however many users share it, so each extra user adds mainly its own cache.
Throughput sets cost. Assume $3.00 per H100-hour, so four GPUs cost $12 an hour, or $0.00333 a second. That is about $34.83 per million tokens at 95.7 tokens a second, and about $0.71 at 4,706. Real prices vary by provider and commitment, and these figures ignore idle time and engine overhead. The cost per million tokens section does this properly.
Averages hide slow requests (performance and latency), so dashboards track percentiles of TTFT and TPOT, such as the 99th percentile, the value that 99% of requests beat. Under load, TTFT rises first, because requests wait in a queue before prefill starts.
Suppose a model's window is 8,192 tokens and a conversation already uses 8,000. The reply can be at most 192 tokens, because the prompt and the reply share one budget. That window is hypothetical. Anthropic's docs define the context window as "all the text a language model can reference when generating a response, including the response itself."
Llama 3.1 has a far larger window. Meta says the 8B and 70B models have "a significantly longer context length of 128K," and the Llama 3 paper says its largest model has "a context window of up to 128K tokens." Taking 128K as 131,072 tokens, a full window limits the 70B model in three ways.
Chat multiplies the cost, because each message resends the whole history. Hugging Face's optimization guide says the cache lets a chat "continuously grow the encoded chat history instead of having to re-encode the chat history again from scratch." Without that reuse, every turn repeats prefill over all earlier turns.
At the limit, Anthropic's docs say that an input which alone exceeds the window fails with an error. On newer models, an input plus max_tokens that exceeds the window is accepted, and generation stops at the window with stop reason model_context_window_exceeded. Code should check the budget before sending.
One conversation takes 8,000 × 327,680 bytes ≈ 2.6 GB, so 148 ÷ 2.62 ≈ 56 fit. Doubling the length halves the count, because the cache grows linearly with the number of tokens.
TTFT is high, and TPOT is normal. Prefill dominates: 50,000 × 140 GFLOP = 7,000 TFLOP, about 3.5 seconds at 50% of the four-H100 peak. Check the queue depth first, then whether many requests share this document, since prefix caching would skip most of that prefill.
Use greedy decoding or a low temperature for extraction, since one answer is right and low temperature keeps the output stable. For names, sample at a higher temperature with top-p, since variety is the goal. Temperature only matters when sampling is on.
The weights are 405 × 10^9 × 2 bytes = 810 GB, more than 8 × 80 = 640 GB, so it does not fit. On 16 H100s, each step reads 810 GB at 16 × 3.35 = 53.6 TB/s, about 15.1 ms, so the ceiling is about 66 tokens a second (1 ÷ 0.0151 ≈ 66).
Compare when the first event leaves the inference server with when the client receives it. If the gap is at a gateway or proxy, that hop is buffering the response. Configure it to forward each event as it arrives.
The request needs 7,900 + 1,000 = 8,900 tokens, so only 8,192 − 7,900 = 292 tokens fit for the reply. Depending on the API, the request is rejected, or it is accepted and stops at the window with stop reason model_context_window_exceeded. Check the budget before sending.
Previous: Design a chat server · Next: GPUs and accelerators