System Design Handbook · llms.txt
Training produces a model once. Serving it runs every day, for every user, and that is where most of the GPU bill goes. An inference engineer makes the serving fast, cheap and reliable. This handbook teaches that job from the bottom up: the arithmetic of a forward pass, the GPU it runs on, the kernels and engines that schedule it, the compression that shrinks it, and the fleet that keeps it up.
The system design handbook on this site explains what an inference platform contains and why. This one goes a level down and asks you to build and measure the pieces. Each chapter links to the system design chapter it builds on instead of repeating it, so read how LLM inference works first if prefill, decode and the KV cache are new.
This chapter covers what the job is, how it differs from its neighbours, what employers ask for, how interviews test it, and how to set up the lab you will use for the rest of the course.
A request for a completion passes through several layers before a single multiplication happens. An inference engineer owns most of them.
The work is measured by a handful of numbers. Time to first token (TTFT) is how long a user waits before anything appears. Inter-token latency (ITL) is the gap between streamed tokens. Throughput is tokens per second per GPU, and cost is dollars per million tokens. The latency metrics section defines them. In a typical week, the work might look like this:
Job titles in this area overlap, and the same title means different things at different companies. The table separates the jobs by what they own.
| Role | Owns | A typical task | Hired for |
|---|---|---|---|
| Inference engineer | Serving engine, model runtime, latency and cost per token | Cut p99 ITL by 30% on a new model | Engines, GPUs, quantization, profiling |
| Kernel or performance engineer | Individual GPU kernels | Write an FP8 attention kernel for a new GPU | CUDA, Triton, CUTLASS, hardware detail |
| ML platform or infrastructure engineer | Clusters, scheduling, deployment, observability | Autoscale model replicas on queue depth | Kubernetes, distributed systems, cloud |
| MLOps engineer | Pipelines from training to production | Automate evaluation and rollout of a new checkpoint | CI/CD, model registries, monitoring |
| Research engineer | Training runs and experiments | Scale a training job to 1,000 GPUs | Training frameworks, parallelism, numerics |
| Applied AI engineer | Products built on models | Build a retrieval assistant on an API | Prompting, RAG, evals, product sense |
The inference engineer sits between the kernel engineer and the platform engineer. You need enough kernel knowledge to know when a kernel is the problem and to write a simple one, and enough platform knowledge to run the engine as a service. Small companies often merge all three jobs into one.
In a sample of inference postings from October 2026 (Together AI, Nebius, Intel, Hedra and Uniphore, listed under Further reading), the same requirements came up again and again:
Senior postings add distributed serving (tensor and expert parallelism, disaggregated prefill and decode) and multi-node operations. The handbook's parts follow this list:
| What postings ask for | Where this handbook teaches it |
|---|---|
| Why a model is slow or fast on given hardware | Part 1, the model as a workload |
| GPU architecture, CUDA, Triton, profiling | Part 2, GPUs and kernels |
| PyTorch compilation, CUDA graphs, C++ extensions | Part 3, the runtime layer |
| Quantization (FP8, INT4, FP4) and quality evaluation | Part 4, compression |
| vLLM, SGLang, TensorRT-LLM internals, speculative decoding | Part 5, serving engines |
| Tensor, pipeline and expert parallelism, disaggregation | Part 6, distributed inference |
| LoRA and QLoRA, serving many adapters | Part 7, fine-tuning for serving |
| Containers, Kubernetes, autoscaling, observability | Part 8, production |
| On-device runtimes (optional) | Part 9, edge inference |
| The interview loop and your portfolio | Part 10, interviews |
Loops differ by company, but most inference interviews draw on five kinds of round:
The labs in this handbook produce material for rounds 1, 2 and 4: working code you wrote, numbers you measured, and write-ups you can talk through.
Every chapter has the same parts:
Labs use three hardware tiers:
| Tag | What it needs | Where to get it |
|---|---|---|
| [Mac] or [any] | A laptop. Apple silicon, any NVIDIA GPU, or a plain CPU | What you have |
| [1 GPU] | One NVIDIA GPU. An L4 or A10G covers most labs; an H100 for FP8 | Hourly rental: RunPod, Lambda, Modal, Vast.ai, Colab |
| [multi-GPU] | 2 to 8 GPUs in one machine | Hourly rental, a few hours per lab |
Batch the GPU labs so one rental covers several, and log what each session cost. A line like "measured on one rented L4 for $3" is a good detail in an interview.
Tier: [any]. This lab runs on any laptop and builds the workspace every later lab uses.
Complete these tasks:
inference-lab, with folders bench/, labs/, notes/ and results/.torch, transformers and numpy.cuda on NVIDIA, mps on Apple silicon.bench/record.py with a function record(name, result) that appends one JSON line to results/results.jsonl, holding the benchmark name, a UTC timestamp, the result, and the environment it ran in (Python, PyTorch, device name, CUDA and driver versions when present, and the git commit).The lab is done when python bench/record.py appends a line naming your device and results.jsonl is committed.
The recorder matters because benchmark numbers without the versions and hardware they came from cannot be compared later. A software upgrade alone can move results a long way: vLLM reported 2.7 times the throughput for version 0.6.0 over 0.5.3, on Llama 3 8B on one H100.
The reference recorder is also available as a file, record.py, to save into your lab repository as bench/record.py. Every Python file the handbook's labs use is in the site's repository under public/labs/infra/, and the scripts that need an NVIDIA GPU are kept apart in public/labs/infra/gpu/.
Set up the repository and the environment:
mkdir inference-lab && cd inference-lab && git init
mkdir bench labs notes results
uv venv --python 3.12 && source .venv/bin/activate
uv pip install torch transformers numpy
python -c "import torch; print(torch.cuda.is_available(), torch.backends.mps.is_available())"
Then write bench/record.py:
# bench/record.py: append one benchmark result, with the machine and software it ran on, to a log.
import datetime
import json
import platform
import subprocess
import sys
def environment() -> dict:
env = {"python": platform.python_version(), "os": platform.platform(), "machine": platform.machine()}
try:
import torch
env["torch"] = torch.__version__
if torch.cuda.is_available():
env["device"] = torch.cuda.get_device_name(0)
env["cuda"] = torch.version.cuda
env["gpu_count"] = torch.cuda.device_count()
elif torch.backends.mps.is_available():
env["device"] = "mps"
except ImportError:
pass
if platform.system() == "Darwin":
env["cpu"] = subprocess.run(["sysctl", "-n", "machdep.cpu.brand_string"],
capture_output=True, text=True).stdout.strip()
try: # NVIDIA driver version, if nvidia-smi exists
env["driver"] = subprocess.run(
["nvidia-smi", "--query-gpu=driver_version", "--format=csv,noheader"],
capture_output=True, text=True, check=True).stdout.split("\n")[0]
except (FileNotFoundError, subprocess.CalledProcessError):
pass
try:
env["git_commit"] = subprocess.run(["git", "rev-parse", "--short", "HEAD"],
capture_output=True, text=True, check=True).stdout.strip()
except (FileNotFoundError, subprocess.CalledProcessError):
pass
return env
def record(name: str, result: dict, path: str = "results/results.jsonl") -> dict:
row = {"name": name, "time": datetime.datetime.now(datetime.timezone.utc).isoformat(timespec="seconds"),
"env": environment(), "result": result}
with open(path, "a") as f:
f.write(json.dumps(row) + "\n")
return row
if __name__ == "__main__":
row = record("smoke-test", {"note": sys.argv[1] if len(sys.argv) > 1 else "setup works"})
print(json.dumps(row, indent=2))
This is the output on an Apple M2 laptop before the first commit, so there is no git_commit yet:
{
"name": "smoke-test",
"time": "2026-10-31T13:39:03+00:00",
"env": {
"python": "3.12.10",
"os": "macOS-26.6.2-arm64-arm-64bit",
"machine": "arm64",
"torch": "2.15.0",
"device": "mps",
"cpu": "Apple M2"
},
"result": {
"note": "setup works"
}
}
On a rented NVIDIA machine the same script adds device (for example "NVIDIA L4"), cuda, gpu_count and driver. Later labs import record and pass their measurements as result.
The router picks a replica, for example the one holding the prompt's prefix in cache. The engine scheduler picks which requests join this step's batch and where their KV cache blocks go. The model runner turns the batch into GPU work, often by replaying a captured CUDA graph. The kernels do the matrix multiplies and attention, and they read weights and KV cache from GPU memory every step.
A kernel engineer makes individual GPU operations fast. A platform engineer runs clusters, deployments and autoscaling. The inference engineer owns the serving engine and the model's runtime between them: batching, memory, quantization, and the latency and cost of each token. The job needs enough kernel knowledge to find and fix a slow kernel, and enough platform knowledge to run the engine as a service.
Serving performance depends on the engine version, PyTorch, CUDA, the driver and the exact GPU. Without them, two results cannot be compared, and a later regression cannot be traced to its cause. An engine upgrade alone can move throughput by a large margin.
Describe a fixed workload (prompt and output lengths, request rate), a baseline measured with warmup and several runs, and the metric that matters, for example p99 ITL at a given throughput. Then describe the change made on its own, the same measurement repeated afterwards, and a quality check if the change could affect outputs. Report the spread, not only the best run.
Next: Models and machines