System Design Handbook · llms.txt
Before touching a profiler, an inference engineer should be able to say roughly how fast a model will run: how long one decode step takes, how many users fit, how long a long prompt makes the first token wait, and what a million tokens cost. These estimates take a few lines of arithmetic, they are usually within a factor of two of reality, and when a measurement disagrees with them the gap tells you where to look.
This chapter builds that arithmetic into a small model of one decode step and one prefill for Llama 3.1 8B, checks the roofline idea against a real machine, and ends with a lab that measures a serving engine against the model.
Before this chapter, read compute versus memory bandwidth for the roofline itself, where the bytes go for the per-operation view, and the numbers in models and machines.
The roofline says an operation takes at least the longer of two times: its FLOPs divided by peak compute, and its bytes divided by memory bandwidth. A model step is a sequence of operations, so its ideal time is the sum over operations:
step time ≈ sum over operations of max(FLOPs ÷ peak, bytes ÷ bandwidth)
Applying the roofline to the whole step at once can mislead, because one part can be compute-bound while another is memory-bound. For a decode step, two groups are enough:
Element-wise operations are small enough to leave out of a first estimate, and the previous chapter's table shows they are under 2% of a decode step.
The roofline's two numbers, peak FLOPs and bandwidth, come from spec sheets, and real hardware reaches only part of them. Measure them first. The probe below times a large copy (pure bandwidth) and a large square matmul (near-peak compute), then sweeps the batch size of a decode-shaped matmul: one 4,096 × 14,336 weight, Llama 3.1 8B's up projection, multiplied by B token vectors. For each B it prints the measured time next to the roofline's prediction from the two measured numbers. You can download it as roofline_probe.py; it runs on Apple silicon, on an NVIDIA GPU or on a CPU.
# roofline_probe.py: measure your machine's bandwidth and matmul peak, then sweep the batch size of a
# decode-shaped matmul (one weight matrix, B token vectors) and compare the times with the roofline.
import statistics
import time
import torch
dev = "cuda" if torch.cuda.is_available() else "mps" if torch.backends.mps.is_available() else "cpu"
dtype = {"cuda": torch.bfloat16, "mps": torch.float16, "cpu": torch.float32}[dev]
elem = torch.tensor([], dtype=dtype).element_size()
def sync():
if dev == "cuda":
torch.cuda.synchronize()
elif dev == "mps":
torch.mps.synchronize()
def timed(fn, reps=20):
"""Median wall time of fn in seconds, after warmup, with the device synchronized."""
for _ in range(3):
fn()
sync()
times = []
for _ in range(reps):
t0 = time.perf_counter()
fn()
sync()
times.append(time.perf_counter() - t0)
return statistics.median(times)
# 1. Bandwidth: copy 256 MB. A copy reads and writes every byte.
src = torch.empty(256 * 2**20 // elem, dtype=dtype, device=dev)
dst = torch.empty_like(src)
bandwidth = 2 * src.numel() * elem / timed(lambda: dst.copy_(src))
# 2. Compute: a large square matmul has plenty of reuse, so it runs near the arithmetic peak.
n = 4096
a = torch.randn(n, n, dtype=dtype, device=dev)
b = torch.randn(n, n, dtype=dtype, device=dev)
peak = 2 * n**3 / timed(lambda: a @ b, reps=10)
ridge = peak / bandwidth
print(f"{dev}, {dtype}: copy bandwidth {bandwidth / 1e9:.0f} GB/s, matmul {peak / 1e12:.2f} TFLOP/s, "
f"ridge {ridge:.0f} FLOPs/byte")
# 3. A decode-shaped matmul: one 4096 x 14336 weight (Llama 3.1 8B's up projection), B tokens.
d_in, d_out = 4096, 14336
w = torch.randn(d_out, d_in, dtype=dtype, device=dev)
print(f"{'batch':>6} {'time ms':>8} {'model ms':>8} {'TFLOP/s':>8} {'GB/s':>6} {'FLOPs/byte':>10}")
for batch in (1, 4, 16, 64, 128, 256, 512, 1024, 2048):
x = torch.randn(batch, d_in, dtype=dtype, device=dev)
t = timed(lambda: x @ w.T)
flops = 2 * batch * d_in * d_out
bytes_moved = (d_out * d_in + batch * d_in + batch * d_out) * elem
model = max(flops / peak, bytes_moved / bandwidth) # the roofline's prediction
print(f"{batch:6d} {t * 1e3:8.3f} {model * 1e3:8.3f} {flops / t / 1e12:8.2f} "
f"{bytes_moved / t / 1e9:6.0f} {flops / bytes_moved:10.1f}")
This is the output on an Apple M2 laptop with the 8-core GPU, running FP16 through PyTorch's MPS backend:
mps, torch.float16: copy bandwidth 84 GB/s, matmul 2.53 TFLOP/s, ridge 30 FLOPs/byte
batch time ms model ms TFLOP/s GB/s FLOPs/byte
1 1.527 1.404 0.08 77 1.0
4 1.585 1.405 0.30 74 4.0
16 2.174 1.411 0.86 54 15.9
64 3.328 2.970 2.26 36 62.7
128 6.156 5.940 2.44 20 123.1
256 11.912 11.879 2.52 11 237.0
512 23.612 23.759 2.55 6 441.1
1024 48.197 47.518 2.50 3 774.9
2048 95.353 95.036 2.52 2 1246.6
Four things stand out in the output:
Run the probe on any machine you rent. On an H100 the shape is the same, with the corner at a far larger batch.
The next script applies the step-time model to Llama 3.1 8B in BF16 on an H100 and on an L4, at 2,000-token and 32,000-token contexts. It caps the batch at what memory holds after the weights and a 10% reserve, and reports the step time, how it splits between weights and KV cache, and the speed per user and in total. It also estimates prefill and the cost per token, which later sections use. The script needs no packages, and you can download it as step_model.py to change the GPU, the model or the context.
# step_model.py: a roofline model of one decode step and one prefill for Llama 3.1 8B in BF16.
# The weight matmuls and the attention reads are timed separately, because batching shares the
# weights across sequences but every sequence reads its own KV cache.
GPUS = { # name: (memory bytes, bandwidth bytes/s, dense BF16 FLOP/s)
"H100 SXM": (80e9, 3.35e12, 989e12),
"L4": (24e9, 300e9, 121e12),
}
MATMUL_PARAMS = 7.50e9 # weights one token multiplies through (model_stats.py)
WEIGHT_BYTES = 2 * MATMUL_PARAMS
KV_PER_TOKEN = 131_072 # bytes: 2 (K, V) x 32 layers x 8 KV heads x 128 x 2 bytes
ATTN_FLOPS_PER_CTX = 4 * 32 * 32 * 128 # 2 matmuls x 2 FLOPs x layers x query heads x head dim
RESERVE = 0.10 # memory kept back for activations and the runtime
def decode_step(gpu, batch, context):
"""Seconds for one decode step: each part takes the longer of its compute and memory time."""
_, bw, peak = GPUS[gpu]
weights = max(batch * 2 * MATMUL_PARAMS / peak, WEIGHT_BYTES / bw)
attention = max(batch * context * ATTN_FLOPS_PER_CTX / peak, batch * context * KV_PER_TOKEN / bw)
return weights, attention
def max_batch(gpu, context):
mem = GPUS[gpu][0]
return int((mem * (1 - RESERVE) - 16.06e9) // (context * KV_PER_TOKEN)) # all 8.03B stored
def prefill_seconds(gpu, prompt, mfu=0.5):
"""Prefill is compute-bound; causal attention scores about half of the prompt x prompt square."""
flops = 2 * prompt * MATMUL_PARAMS + ATTN_FLOPS_PER_CTX * prompt * prompt / 2
return flops / (GPUS[gpu][2] * mfu)
if __name__ == "__main__":
for gpu in GPUS:
for context in (2_000, 32_000):
cap = max_batch(gpu, context)
print(f"{gpu}, {context:,}-token contexts (memory fits batch {cap}):")
for batch in (1, 8, 32, 64, 128, 256):
if batch > cap:
break
w, a = decode_step(gpu, batch, context)
step = w + a
print(f" batch {batch:3d}: step {step * 1e3:6.2f} ms (weights {w / step:4.0%}, "
f"KV {a / step:4.0%}), {1 / step:5.0f} tok/s per user, {batch / step:6.0f} total")
for gpu in GPUS:
for prompt in (1_000, 8_000, 32_000):
print(f"{gpu}: prefill {prompt:6,} tokens at 50% MFU = {prefill_seconds(gpu, prompt) * 1e3:7.1f} ms")
# Price per million tokens on one H100 at an assumed $3.00 an hour.
hour = 3.00
prefill_tok_s = 8_000 / prefill_seconds("H100 SXM", 8_000)
w, a = decode_step("H100 SXM", 64, 2_000)
decode_tok_s = 64 / (w + a)
for name, rate in (("input (prefill, 8K prompts)", prefill_tok_s), ("output (decode, batch 64)", decode_tok_s)):
print(f"{name}: {rate:9,.0f} tok/s -> ${hour / 3600 / rate * 1e6:.3f} per million tokens")
H100 SXM, 2,000-token contexts (memory fits batch 213):
batch 1: step 4.56 ms (weights 98%, KV 2%), 219 tok/s per user, 219 total
batch 8: step 5.10 ms (weights 88%, KV 12%), 196 tok/s per user, 1568 total
batch 32: step 6.98 ms (weights 64%, KV 36%), 143 tok/s per user, 4583 total
batch 64: step 9.49 ms (weights 47%, KV 53%), 105 tok/s per user, 6747 total
batch 128: step 14.49 ms (weights 31%, KV 69%), 69 tok/s per user, 8831 total
H100 SXM, 32,000-token contexts (memory fits batch 13):
batch 1: step 5.73 ms (weights 78%, KV 22%), 175 tok/s per user, 175 total
batch 8: step 14.49 ms (weights 31%, KV 69%), 69 tok/s per user, 552 total
L4, 2,000-token contexts (memory fits batch 21):
batch 1: step 50.87 ms (weights 98%, KV 2%), 20 tok/s per user, 20 total
batch 8: step 56.99 ms (weights 88%, KV 12%), 18 tok/s per user, 140 total
L4, 32,000-token contexts (memory fits batch 1):
batch 1: step 63.98 ms (weights 78%, KV 22%), 16 tok/s per user, 16 total
H100 SXM: prefill 1,000 tokens at 50% MFU = 30.9 ms
H100 SXM: prefill 8,000 tokens at 50% MFU = 276.6 ms
H100 SXM: prefill 32,000 tokens at 50% MFU = 1513.5 ms
L4: prefill 1,000 tokens at 50% MFU = 252.3 ms
L4: prefill 8,000 tokens at 50% MFU = 2260.8 ms
L4: prefill 32,000 tokens at 50% MFU = 12370.8 ms
input (prefill, 8K prompts): 28,923 tok/s -> $0.029 per million tokens
output (decode, batch 64): 6,747 tok/s -> $0.124 per million tokens
Four lessons come out of the decode rows.
The explorer below runs the same model for six chips and three Llama sizes. Try these experiments:
Every decode step emits one token for each sequence in the batch, so the inter-token latency (ITL) a user sees is the step time, and total throughput is batch ÷ step time. Raising the batch raises throughput and ITL together. The product decides where to stop.
A latency target turns that into a maximum batch. Suppose users need at least 50 tokens a second, an ITL of 20 ms. From the model, at 2,000-token contexts on an H100 the step reaches 20 ms at about batch 200 (4.5 ms of weights plus 200 × 2,000 × 131,072 bytes ÷ 3.35 TB/s ≈ 15.6 ms of cache). The batch the target allows, and the batch memory allows (213), happen to be close here. With longer contexts, memory runs out first; with a stricter target, latency does.
This is the trade-off continuous batching manages step by step (batching requests), and it is why a serving benchmark must report throughput at a stated latency, never alone.
Prefill is compute-bound, so its time is FLOPs ÷ (peak × MFU), where MFU (model FLOPs utilization) is the share of peak a real kernel sustains. The script assumes 50%, a deliberately round guess; measure your own in the lab.
Time to first token = time waiting in the queue + prefill of the prompt + the first decode step.
The prefill rows show two things:
At an assumed $3.00 per H100-hour (the same assumption as GPUs and accelerators), the model gives about $0.03 per million input tokens (prefill of 8K-token prompts) and $0.12 per million output tokens (decode at batch 64), at full utilization. Output costs about 4 times as much as input, because a prefill reads the weights once for thousands of tokens and a decode step reads them once for each token of the batch.
This is also why many hosted model APIs price output tokens several times higher than input tokens: the hardware cost really is asymmetric. Real prices also cover idle capacity, redundancy and margin. At 30% average utilization, every figure above roughly triples.
Two ratios say how close a real system comes to the model:
For example, the lab below measured vLLM serving Llama 3.1 8B at batch 1 on an H100 at 152 tokens a second. Each 6.56 ms step reads about 15.0 GB of weights plus 0.3 GB of cache, so MBU = 15.3 GB ÷ 6.56 ms ÷ 3.35 TB/s ≈ 69%. The model's 219 tokens a second would be MBU 100%. Measured MBU tells you how much there is left to gain.
When a measurement lands below the model, these are the usual reasons, roughly in order of how often they matter for decode:
Tier: [any] for part A, [1 GPU] for part B. Part A runs on your own machine, and part B needs one rented NVIDIA GPU.
Part A, your machine's roofline. Run roofline_probe.py above on your laptop and, during your next GPU rental, on an L4 or H100. Record both with bench/record.py. Plot measured time against batch on log-log axes with the model's line on top, and mark the ridge point. Part A is done when you can say, for each machine, at what batch the decode-shaped matmul stops being free, and where the model and measurement disagree most.
Part B, a real engine. On one rented GPU, serve Llama 3.1 8B in BF16 with vLLM and measure decode step time at batch 1, 8, 32 and 64, with 2,000-token prompts. Then complete these steps:
To isolate decode, time a run that generates 1 token and a run that generates 257 tokens from the same prompts. The difference is 256 decode steps, without the prefill.
Part B is done when the table and a paragraph of explanation are in notes/ and the raw results are in results.jsonl.
Part A has no single right answer; your laptop's numbers are the answer. Check that the flat region at small batches matches your copy bandwidth, and that the large-batch region matches your square-matmul peak.
For part B, the script gpu/decode_sweep.py loads the model in vLLM once and times the two runs per batch with vLLM's own generate, using the same settings as vllm bench latency: random prompt tokens, ignore_eos so that every run generates exactly the requested number of tokens, and the median of 10 timed runs. It uses unsloth/Meta-Llama-3.1-8B-Instruct, an ungated copy of Meta's weights, and then repeats batches 1 and 8 with enforce_eager=True. The heart of it is the subtraction:
def run_seconds(prompts, output_len, warmup=3, iters=10):
params = SamplingParams(temperature=1.0, top_p=1.0, ignore_eos=True, max_tokens=output_len)
... # warm up, then return the median of `iters` timed runs
step = (run_seconds(prompts, STEPS + 1) - run_seconds(prompts, 1)) / STEPS # STEPS = 256
context = PROMPT + STEPS / 2 # average context during the 256 steps
bytes_per_step = WEIGHT_BYTES + batch * context * KV_PER_TOKEN
mbu = bytes_per_step / step / BANDWIDTH
On Modal it runs with modal run modal_run.py --script decode_sweep.py --gpu 'H100!' --vllm. Its output on an H100:
NVIDIA H100 80GB HBM3, vLLM 0.32.0, PyTorch 2.13.0+cu130
unsloth/Meta-Llama-3.1-8B-Instruct, BF16, 2000-token prompts, 256 decode steps timed (median of 10 runs)
Model loaded and ready in 195 s with CUDA graphs and torch.compile, 18 s with enforce_eager
batch step model MBU tok/s per user tok/s total
default 1 6.56 ms 4.56 ms 69% 152 152
default 8 7.27 ms 5.14 ms 71% 138 1101
default 32 9.67 ms 7.14 ms 74% 103 3310
default 64 12.87 ms 9.81 ms 76% 78 4971
enforce_eager 1 14.90 ms 4.56 ms 31% 67 67
enforce_eager 8 14.69 ms 5.14 ms 35% 68 545
The 195 seconds of the first load include downloading the 16 GB of weights, compiling the model with torch.compile and capturing its CUDA graphs; the second load found the weights already on disk and compiled nothing. How to read the table:
enforce_eager=True, which in vLLM 0.32 turns off both, batch 1 took 14.90 ms instead of 6.56 ms, and batch 8 took no longer than batch 1, the signature of a step limited by launching kernels from the CPU rather than by the GPU's work. This is the effect that chapter 3.1 measures kernel by kernel.max_num_seqs or max_num_batched_tokens can cap the real batch, and gpu_memory_utilization can leave too little room for the cache. An MBU above 100% means the byte count is wrong, for example a KV cache in FP8 when you assumed BF16.The matmul weights take about 7.5 GB in FP8. 7.5 GB ÷ 3.35 TB/s ≈ 2.24 ms per step, so about 450 tokens a second, twice the BF16 ceiling, before overheads and KV cache reads.
The matmul is memory-bound: its time is the time to read the weight matrix, and extra token vectors add almost no bytes. It stops near the ridge point, where the intensity (about equal to the batch for this shape) reaches peak FLOPs ÷ bandwidth, about 30 on the M2. The measurement shows the time already rising at batch 16 and clearly compute-bound by 128.
They would at batch 295, the ridge point. Memory runs out first: after the 16 GB of weights and a 10% reserve, about 56 GB of cache holds 213 sequences of 2,000 tokens. The KV cache, not compute, caps the batch.
The prompt's prefill takes over a second on an H100, and if the engine runs it as one piece, the decode batch gets no steps meanwhile, so every stream pauses. Chunked prefill splits the prompt into pieces that run alongside decode steps, trading a slightly later first token for the long prompt against smooth ITL for everyone else. Separating prefill and decode onto different GPUs is the larger-scale fix.
Bytes per step = 15.0 GB of weights + 32 × 2,000 × 131,072 ≈ 8.4 GB of cache = 23.4 GB. Achieved bandwidth = 23.4 GB ÷ 0.010 s = 2.34 TB/s. MBU = 2.34 ÷ 3.35 ≈ 70%.
Different parts of a step sit on different sides of the ridge point. At large batch, the weight matmuls may be compute-bound while attention, at an intensity of 4, is memory-bound. One roofline on the totals would average them and predict a shorter step than either part allows. Summing per-part times keeps each part on its own limit.
vllm bench latency reference documents the flags this chapter's lab uses, and its benchmarking CLI guide covers vllm bench serve and the other benchmark commands that Part 8 uses.Previous: The transformer, seen by the hardware · Next: Mixture-of-experts models