System Design Handbook · llms.txt

AI Infrastructure Engineer Handbook

Inference from kernels to clusters: how models run on GPUs, how serving engines schedule them, and how to make them fast and cheap. Each chapter has a lab.

Orientation

What an inference engineer does, how the job is hired for, and the models and machines the labs use.

  • The inference engineer: Training produces a model once. Serving it runs every day, for every user, and that is where most of the GPU bill goes. An inference engineer makes the serving fast, cheap and…
  • Models and machines: Numbers only make sense against a fixed model and a fixed machine. This handbook uses three models and three machines throughout, so a figure in Part 5 can be checked against one…

The model as a workload

Where the FLOPs and bytes of a transformer go, the arithmetic that predicts speed, latency and cost, and what mixture-of-experts models change.

  • The transformer, seen by the hardware: To a GPU, a transformer is a list of matrix multiplies with some element-wise work in between. Which of those operations takes the time depends on how many tokens go through at…
  • The arithmetic of inference: Before touching a profiler, an inference engineer should be able to say roughly how fast a model will run: how long one decode step takes, how many users fit, how long a long…
  • Mixture-of-experts models: A mixture-of-experts (MoE) model replaces each layer's MLP with many smaller MLPs, called experts, and a router that sends each token to a few of them. The model stores far more…

GPUs and kernels

How a GPU runs threads and moves memory, how to write kernels in Triton and read them in CUDA, FlashAttention, and profiling.

  • How a GPU executes code: Part 1 treated a GPU as two numbers, peak FLOPs and memory bandwidth, and that was enough to predict how fast a model runs. Writing or fixing a kernel needs a closer look. This…
  • Writing kernels in Triton: Triton is a language for writing GPU kernels in Python. Instead of describing what each of thousands of threads does, a Triton kernel describes what one program does to a block of…
  • CUDA for reading and debugging: Most inference engineers write more Triton than CUDA, but they read CUDA every week. The fastest kernels in vLLM, SGLang, FlashAttention and TensorRT-LLM are CUDA C++, and when…
  • FlashAttention from the ground up: Attention is the one part of a transformer whose cost grows with the square of the sequence. Computed the obvious way, it writes an N × N matrix of scores to GPU memory, reads it…
  • Profiling and benchmarking: Every claim in this part of the handbook rests on a measurement, and a measurement is only as good as its method. This chapter covers both halves of that work. Benchmarking asks…

The runtime layer

What eager PyTorch costs, CUDA graphs, torch.compile and the caching allocator; C++ bindings and the GIL; and ahead-of-time compilers and runtimes.

  • PyTorch execution for inference: Chapter 2.5 profiled one decode step of Llama 3.2 1B on an H100 and found the GPU idle for most of it: hundreds of tiny kernels, each launched separately by Python, and a GPU that…
  • Native code and Python bindings: Every major inference engine is a Python program on top of native code. Python holds the parts that change often, such as the scheduler's policy, the API server and the model…
  • Compilers and graph runtimes: Chapter 3.1 compiled a model just in time: torch.compile traced the decode step inside the serving process, on its first call, and the process kept the result in memory. The other…

Compression

Number formats from FP32 to NVFP4, the arithmetic of quantization, the algorithms that produce quantized checkpoints, how to measure what they cost in quality, and when pruning or distillation is worth it instead.

  • Number formats: Every weight, activation and cache entry a model touches is stored in some number format, and the format decides three things at once: how many bytes the hardware must move, which…
  • Quantization math: Chapter 4.1 showed what the small number formats can hold. This chapter covers how a model's weights and activations are mapped onto them: the scale and zero-point that turn…
  • Post-training quantization algorithms: Rounding every weight to its nearest grid point, the method chapter 4.2 used, is called round-to-nearest (RTN). It works well at 8 bits and poorly at 4: on SmolLM2-135M, INT4 with…
  • Measuring quality loss: A quantized model is only worth serving if it still does its job, and "the perplexity went up by 2%" does not say whether it does. This chapter covers how to measure what…
  • Pruning and distillation: Quantization keeps every weight and stores it in fewer bits. The two other ways to make a model cheaper remove work instead. Pruning sets weights to zero, either anywhere or in a…

Serving engines

An inference engine built from scratch, with a paged KV cache, continuous batching under a token budget, chunked prefill, preemption and an OpenAI-compatible server, measured against vLLM. Then vLLM itself: one request traced through its processes, its scheduler and block pool read against yours, and its main settings measured. Then prefix caching: the radix tree of SGLang next to the hashed blocks of vLLM, and both engines measured on a multi-turn chat workload. Then speculative decoding: the acceptance rule, drafters from prompt lookup to EAGLE-3, and vLLM measured with six of them from 1 request in flight to 128. Then the last step of every step: the sampler, what top-p costs and how engines avoid sorting, and structured output, from a token mask built by hand to XGrammar inside vLLM.

  • Build a mini inference engine: Inference serving engines described what an engine does: it batches requests at every step, keeps their KV caches in fixed-size blocks, splits long prompts into chunks, and…
  • Inside vLLM: Chapter 5.1 built an engine in about 650 lines. vLLM is the same loop with three years of production requirements attached: the vllm/v1 folder alone holds about 186,000 lines of…
  • SGLang and prefix caching: Chapter 5.2 ended with one measurement of prefix caching: 63 requests that shared a 3,000-token prefix finished 12.7 times sooner when vLLM could reuse the prefix's keys and…
  • Speculative decoding: Every technique in this part so far has made the server do the same work more efficiently. Speculative decoding is different in kind: it makes the server do more work in order to…
  • Structured output and sampling: The forward pass ends with a vector of logits, one score for every token in the vocabulary, and the engine still has to turn that vector into one token. Two pieces of an engine do…