System Design Handbook · llms.txt
C++ for AI Infrastructure Engineers
From your first CMake build to CUDA kernels, for engineers who know Python: memory, ownership, performance, concurrency and the GPU, ending with an LLM inference engine you write yourself.
The ground rules
How C++ turns source files into a program, and the syntax that differs from Python, then memory, pointers and undefined behavior.
- The toolchain and the compilation model: Sooner or later every AI infrastructure engineer types cmake --build on someone else's project and gets a wall of red text: llama.cpp with a new backend switched on, a PyTorch…
- C++ syntax for Python programmers: You already know what a loop is, what a function is and how an if works, so this chapter does not teach programming again. It teaches the differences, and there are two kinds. The…
- Memory: stack, heap and lifetime: Here is a number that surprises most Python programmers. A Python list of a million floats, built with something like [float(i) for i in range(1_000_000)], takes about 32 MB on a…
- Pointers, references and const: Open any CUDA kernel, any BLAS routine or any PyTorch custom operator, and the first line looks something like this:
- Undefined behavior: Here is a bug that every ML systems engineer meets sooner or later. A custom kernel computes the mean of each row of an activation matrix. It compiles without a warning, it runs…
- Project A: a log analyzer in Python and C++: It is the middle of the night, and the inference service you run has an alert: users say chat responses feel slow. The dashboards are down, but every server writes one line per…
Ownership and abstraction
Classes whose destructors release what they own, and moves that hand large data from one owner to the next without copying it.
- Classes and RAII: Imagine a model server that loads LoRA adapters from disk when a request asks for one. The loader opens the adapter file, reads a small header, and if the adapter was trained for…
- Copy and move semantics: Here is a bug that costs nothing to write and a great deal to run. A small C++ inference engine walks through a model's layers for every token it generates: