System Design Handbook · llms.txt
Every claim in this part of the handbook rests on a measurement, and a measurement is only as good as its method. This chapter covers both halves of that work. Benchmarking asks how fast something is, and it is easy to get a precise but wrong answer. Profiling asks where the time goes, and it needs the right tool for the question: the PyTorch profiler for a whole model step, Nsight Systems for a timeline of a program, and Nsight Compute for the counters inside one kernel. The lab profiles one decode step of Llama 3.2 1B on an H100 and finds that the GPU spends most of the step waiting.
Before this chapter, read how a GPU executes code and the arithmetic of inference, whose step-time model is what a profile is compared against.
The same operation can produce very different numbers depending on how it is timed. Two measurements in this part show it. Chapter 2.1's probe timed an 8,192-square BF16 matmul on an H100 at 803 TFLOP/s, and chapter 2.2's script timed the same cuBLAS call on another H100 at 678 TFLOP/s. Neither is wrong. The probe reported the fastest of 20 back-to-back runs, which take under 30 ms, while Triton's do_bench, according to its source, repeats the call for about 100 ms by default, clears the L2 cache before every run and returns the mean. The longer measurement is more likely to reach the GPU's power limit, described below. Neither number means anything without its method.
A sound benchmark follows a few rules:
torch.cuda.synchronize() measures only the launch. CUDA events, recorded on the GPU's own stream, measure the GPU's time directly.do_bench does. A kernel that really does reuse data across calls can be timed warm, but say so.The lab measured both of the last effects on Modal H100s. While a long run of BF16 matmuls was in flight, nvidia-smi reported the GPU at its 700 W power limit, with the SM clock down from 1,980 MHz to about 1,400 MHz:
4. SM clock and power while 3,000 BF16 8192-square matmuls run back to back (about 4 s on an H100)
idle: 1980 MHz, 1980 MHz, 134.80 W, 700.00 W, 28
busy 1: 1395 MHz, 1980 MHz, 699.86 W, 700.00 W, 45
busy 2: 1455 MHz, 1980 MHz, 695.89 W, 700.00 W, 47
busy 3: 1905 MHz, 1980 MHz, 611.18 W, 700.00 W, 35
busy 4: 1980 MHz, 1980 MHz, 449.95 W, 700.00 W, 33
busy 5: 1980 MHz, 1980 MHz, 197.92 W, 700.00 W, 32
busy 6: 1980 MHz, 1980 MHz, 119.68 W, 700.00 W, 32
The columns are the current SM clock, the maximum SM clock, the power draw, the power limit and the temperature in degrees Celsius. The samples probably begin near the end of the run: Python can queue only a limited number of kernels ahead of the GPU, so the loop that queues the matmuls returns when most of them have already run. The first two samples catch the GPU at its power limit, and the later ones show the clock recovering and the power falling as the last matmuls finish. This is the explanation chapter 2.1 left open for the BF16 rate dipping at the largest matmul: a long, dense run of matmuls reaches the power limit, and the GPU lowers its clock to stay within it. A short benchmark that never reaches the limit will report a higher rate than a long production run achieves.
The host mattered too. The same script, run three times on Modal H100s, measured a batch-1 decode step of Llama 3.2 1B at 15.68 ms, 7.49 ms and 16.12 ms, and chapter 3.1's lab measured the same step at 10.30 ms. The code that times the step did not change between the runs, and the third run reserved four CPU cores. The step is limited by how fast the CPU launches kernels, as the next section shows, so a different host CPU or a busier neighbour changed the result by a factor of two, and reserving cores did not remove the difference. A benchmark report should name the GPU, the driver, the library versions and, for anything launch-bound, the CPU. The record.py helper from chapter 0.1 captures most of these.
torch.profiler records every PyTorch operation on the CPU and every kernel on the GPU, and links them, so it answers the first question to ask of a slow model: is the time on the CPU, on the GPU, or between them? The lab's profile_decode.py runs Llama 3.2 1B in BF16 at batch 1 with a 512-token prompt, then profiles one decode step on an H100, run with four reserved CPU cores (--cpu 4):
NVIDIA H100 80GB HBM3, PyTorch 2.15.0+cu132; unsloth/Llama-3.2-1B-Instruct: 2.47 GB of weights
1. Decode step at batch 1: 16.12 ms wall time (62 tokens/s); reading the weights once at the spec bandwidth would take 0.74 ms
2. One profiled step: 694 GPU kernels, 549 kernel launches from the CPU; GPU busy 2.24 ms of a 24.95 ms span (9%); longest gap between kernels 198 us
The model predicts 0.74 ms, the time to read 2.47 GB of weights at 3.35 TB/s, and the step takes 16.12 ms on this host, more than twenty times longer. The profile shows why. The step runs 694 kernels, which add up to 2.24 ms of GPU work, so even without the profiler the GPU is idle for about 86% of the step. The median kernel in the trace runs for 1.7 µs, shorter than the time the CPU takes to launch the next one. The profiler itself slows the CPU side further, which is why the profiled span is 25 ms rather than 16; the GPU times inside it are still accurate.
A profile of one step records one-time costs along with recurring ones, so look at its longest gap before trusting it. The first version of this lab profiled a step at a KV cache length that no earlier step had used, and its trace showed a single 40.8 ms gap just before the first attention kernel. That gap was cuDNN building its attention plan for the new length, which chapter 3.1 explains. The script now profiles a step at a length the timing loop has already used, and its longest gap is 198 µs.
The profiler's table of the top operations by GPU time shows what the 2.24 ms consist of:
Name Self CUDA Self CUDA % CUDA total CUDA time avg # of Calls
aten::linear 0.000us 0.00% 1.246ms 11.023us 113
aten::mm 1.246ms 55.54% 1.246ms 11.023us 113
nvjet_sm90_tst_64x8_64x16_2x1_v_bz_TNT 447.634us 19.96% 447.634us 13.989us 32
aten::mul 266.418us 11.88% 266.418us 1.788us 149
The table above keeps the columns that matter here; the script prints all of them. The 113 matrix multiplications take 55% of the GPU's time, in cuBLAS kernels whose names begin with nvjet. They move the 2.47 GB of weights in 1.25 ms, about 2 TB/s or 59% of the spec bandwidth, which is reasonable for matmuls with a single row of activations. The 149 element-wise multiplications, from RMSNorm, the rotary embeddings and the MLP's gating, take another 12%, at under 2 µs each. Grouping the trace's kernels by type gives 433 element-wise kernels, 65 concatenations, which come from the rotary embedding's rotate-half step and from appending the new token's keys and values to the KV cache by copying it, 50 reductions and 32 kernels of cuDNN's fused attention, which PyTorch's scaled_dot_product_attention chose for this step.
The widget below draws the recorded trace. The top row is the CPU's launch calls and the bottom row is the GPU's kernels; zoom in to see the gaps:
The diagnosis is that this step is limited by launch overhead, not by the GPU's memory or arithmetic. The fixes are the subject of Part 3: CUDA graphs record the whole sequence of kernels once and replay it with a single launch, and torch.compile fuses the small element-wise kernels into fewer, larger ones. Serving engines such as vLLM do both, which is why they decode far faster than a plain Hugging Face forward pass on the same GPU.
Nsight Systems records a timeline of a whole program: CPU threads, CUDA API calls, kernels, memory copies, and NVTX ranges that the program marks itself. It is the tool for questions about how the parts fit together, such as whether the GPU waits for the CPU, whether data copies overlap with compute, or whether two streams really run at the same time. A typical command and summary report look like this:
nsys profile --trace cuda,nvtx,osrt -o decode python my_script.py
nsys stats --report cuda_gpu_kern_sum --report cuda_api_sum decode.nsys-rep
The first command writes a .nsys-rep file to open in the Nsight Systems GUI, and the second prints tables of kernel time and API time. The lab's nsys_decode.py wraps both around the same Llama 3.2 1B decode step. It marks each step with an NVTX range, records only 20 steps between cudaProfilerStart and cudaProfilerStop (the --capture-range cudaProfilerApi option), and prints three summaries. On Modal it needs the toolkit image, whose Nsight Systems importer needs the libdw1 system package; without it, nsys records the raw trace but fails to convert it, with an unhelpful "Importer error status: An unknown error occurred". On an H100 with four reserved cores:
NVTX ranges: 20 decode steps, 16.85 ms each on average (CPU time inside the range)
GPU kernels: 13880 launches (694 per step), 2.22 ms of kernel time per step
share count mean kernel
19.7% 640 13.7 us nvjet_sm90_tst_64x8_64x16_2x1_v_bz_TNT
9.9% 320 13.7 us nvjet_sm90_tst_64x8_64x16_2x1_v_bz_splitK_TNT
8.6% 640 6.0 us nvjet_sm90_tst_64x8_64x16_4x1_v_bz_TNT
7.9% 20 174.6 us nvjet_sm90_tst_384x8_64x4_2x1_v_bz_TNT
7.8% 640 5.4 us nvjet_sm90_tst_8x64_64x16_4x1_v_bz_TNN
6.5% 1280 2.3 us void at::native::elementwise_kernel<(int)128, (int)4, void at::native:
3.7% 1280 1.3 us void at::native::vectorized_elementwise_kernel<(int)8, at::native::CUD
3.4% 660 2.3 us void at::native::reduce_kernel<(int)512, (int)1, at::native::ReduceOp<
CUDA API calls on the CPU, by total time:
share count mean call
71.5% 10660 5.6 us cudaLaunchKernel
21.7% 2900 6.3 us cuLaunchKernelEx
2.3% 320 6.1 us cudaLaunchKernelExC_v11060
2.0% 10660 0.2 us cuKernelGetName
1.0% 4520 0.2 us cuKernelGetAttribute
0.5% 640 0.7 us cuKernelSetAttribute
The kernel table agrees with the PyTorch profiler: 694 kernels and about 2.2 ms of GPU work per step, led by the same cuBLAS matmuls. The API table adds the CPU's side. Each step makes 694 launch calls of about 6 µs each, roughly 4 ms of launching per 16.85 ms step, and the rest of the CPU's time goes to Python and PyTorch's own dispatch before each launch. Every one of those microseconds is time the GPU spends waiting, which is the cost that CUDA graphs remove.
The same tool also shows one-time costs. Recording the script's first pass instead, where every step meets a new cache length, gave steps of 48.6 ms on average, and the API table gained one cuLibraryLoadData call and one cuKernelGetFunction call in every step, the second taking about 1 ms each time: most likely cuDNN loading the attention kernel it had just built for that length. That is why the script records a second pass over lengths it has already seen.
Nsight Compute replays a kernel several times and reads the GPU's hardware counters, so it answers why one kernel is slow: whether it is memory-bound or compute-bound, how well its loads are coalesced, how full its SMs are, and what its warps are waiting for. It worked inside a Modal container on an A10 once told not to lock the GPU's clocks, with the option --clock-control none, at the cost of somewhat less consistent results.
The lab's ncu_strided.py runs the same element-wise kernel twice over 16 million float32 values, once reading them contiguously and once reading every eighth value, and asks Nsight Compute for four metrics:
| Metric | Contiguous | Every 8th value |
|---|---|---|
Load requests (l1tex__t_requests_pipe_lsu_mem_global_op_ld.sum) | 131,072 | 524,288 |
Sectors loaded (l1tex__t_sectors_pipe_lsu_mem_global_op_ld.sum) | 2,097,152 | 16,777,216 |
| Sectors per request | 16 | 32 |
| DRAM throughput, share of peak | 92% | 95% |
| Kernel time | 278 µs | 1,190 µs |
These counters show chapter 2.1's coalescing rule directly. The contiguous kernel is vectorized, so each thread loads 16 bytes and each warp request covers 512 bytes, exactly 16 sectors. The strided kernel's warp requests touch 32 different sectors each, the worst case, and it loads eight times as many sectors for the same useful data. Both kernels keep DRAM almost fully busy, so the strided one is not idle; it is busy moving bytes it throws away. It takes 4.3 times as long rather than eight because both kernels also write the same 64 MiB of output: the contiguous kernel moves about 128 MiB in total and the strided one about 576 MiB, 4.5 times as much. A kernel that shows high DRAM throughput is not necessarily efficient, which is why sectors per request is the counter to check first for a memory-bound kernel.
The counter names come from Nsight Compute's Kernel Profiling Guide. Running ncu --set full on a kernel collects most sections at once, including the memory workload analysis and the warp stall reasons, and the GUI shows them with explanations.
A profile is useful when it leads to a decision. Most performance problems in inference fall into one of three cases, and each has its own signature and its own fixes:
| What the profile shows | Diagnosis | Typical fixes |
|---|---|---|
| GPU idle between many short kernels; CPU busy launching | Launch overhead, as in the decode step above | CUDA graphs, torch.compile, kernel fusion, larger batches |
| Long kernels with high DRAM throughput and low compute use | Memory-bound | Fewer bytes: fusion, lower precision weights or KV cache, better access patterns |
| Long kernels with high tensor-core use | Compute-bound | Lower precision such as FP8, fewer FLOPs, or more GPUs |
The roofline from Part 1 tells you which case to expect before you profile, and the profile tells you whether reality agrees. When they disagree, as with the decode step that the roofline put at 0.74 ms and the measurement found at 16.12 ms, the gap is the most useful thing to explain.
Tier: [1 GPU]. The profiling run took 37 seconds of H100 time on Modal, about 4 cents, including a 2.5 GB model download. The Nsight Systems run took 28 seconds of H100 time, and the Nsight Compute run 8 seconds of A10 time, under a cent.
The scripts are ready to download:
decode_trace.json, and samples the GPU's clock and power under load. Run it with modal run modal_run.py --script profile_decode.py --gpu 'H100!' --cpu 4.modal run modal_run.py --script nsys_decode.py --gpu 'H100!' --toolkit --cpu 4.modal run modal_run.py --script ncu_strided.py --gpu A10 --toolkit. Modal's A10 type may give you an A10 or the closely related A10G.Part A, a decode step. Profile one batch-1 decode step of Llama 3.2 1B with torch.profiler. Report the step time, the number of kernels, the GPU's busy time, and the top kernels. Compare the step time with the roofline estimate from chapter 1.2.
Part B, one kernel's counters. Use Nsight Compute to measure sectors per request and DRAM throughput for a contiguous and a strided read, and connect the numbers to the coalescing rule from chapter 2.1.
Part C, a findings memo. Write one page that a teammate could act on: what you measured, how, what limits the step, and what you would try first, with the expected gain. The lab is done when the memo states a diagnosis supported by at least two numbers from your profiles.
Parts A and B are the measurements printed in this chapter. A memo for Part C, written from them, could read as follows.
Decode step of Llama 3.2 1B at batch 1 on one H100: findings.
What we measured. With Hugging Face transformers' standard forward pass in BF16, one decode step after a 512-token prompt takes 16.12 ms of wall time, or 62 tokens a second. The roofline estimate for reading the 2.47 GB of weights once at 3.35 TB/s is 0.74 ms. Earlier runs of the same script on other Modal H100s measured 15.68 ms and 7.49 ms, so results depend on the host and must be reported with it.
Where the time goes. The PyTorch profiler records 694 GPU kernels per step, adding up to 2.24 ms of GPU work, so the GPU is idle for about 86% of the step. The median kernel runs for 1.7 µs. Matmuls take 55% of the GPU time and reach about 2 TB/s; 433 element-wise kernels and 65 KV-cache concatenations make up most of the rest.
Diagnosis. The step is limited by kernel launch overhead on the CPU. It is not limited by the GPU's memory bandwidth or its arithmetic.
What to try first. First switch to a static KV cache (in transformers, cache_implementation="static"), which preallocates the cache and writes each new token in place, so every decode step has the same shapes. Then capture the step in a CUDA graph, for example with torch.compile(mode="reduce-overhead"), which replays all kernels with one launch. That should bring the step close to the GPU's 2.24 ms of work, several times faster. Fusing the element-wise kernels, which torch.compile also does, should then cut the GPU time itself towards the 0.74 ms floor. Chapter 3.1's lab measures these steps.
The first script reported the fastest of 20 runs, while triton.testing.do_bench returns the mean of many runs, which includes the slower ones. And do_bench repeats the call for about 100 ms by default, while the probe's 20 runs take under 30 ms, so the longer measurement is more likely to reach the power limit described in this chapter and run at a lower clock. do_bench also clears the L2 cache before every run, but that matters little here: each 8,192-square BF16 operand is 128 MiB, far larger than the H100's 50 MB L2, so back-to-back runs find little of their data still cached. The two scripts also ran on different H100s, whose clocks and power states can differ.
The kernel durations are measured on the GPU and are reliable, so 2.24 ms is the GPU's real work. The profiler adds CPU overhead to every operation, which stretches the gaps between kernels, so the 25 ms span overstates the step. The step time should come from an unprofiled, synchronised measurement, 16.12 ms here. Together they show that the GPU is idle for about 86% of the step.
The GPU reached its power limit and lowered its clock from 1,980 MHz to stay within it, so its arithmetic rate fell by almost 30% while the limit applied. A short benchmark that ends before the power limit lowers the clock will report a higher rate than a long production run. Record clocks and power during benchmarks, and run them long enough to reach a steady state.
DRAM throughput counts every byte moved, useful or not. The strided read loads 32 sectors per warp request where the contiguous one loads 16, and it needs four times as many requests, so it moves eight times as many sectors for the same useful values. The memory system is just as busy, but most of what it moves is discarded.
For idle gaps between kernels, Nsight Systems or the PyTorch profiler's timeline, since the answer lies in how CPU and GPU work interleave. For one slow kernel, Nsight Compute, which reads the kernel's hardware counters. For which PyTorch operation launches the most kernels, the PyTorch profiler's operator table, which maps kernels back to the operations that launched them.
torch.profiler, its operator tables and the trace export used in the lab.nsys stats reports.Previous: FlashAttention from the ground up · Next: PyTorch execution for inference