{"id":"cpp-memory","title":"Memory: stack, heap and lifetime","sections":[{"id":"overview","title":"Overview","html":"<p>Here is a number that surprises most Python programmers. A Python list of a million floats, built with something like <code>[float(i) for i in range(1_000_000)]</code>, takes about 32 MB on a 64-bit CPython: each float is a separate 24-byte object somewhere in memory, and the list holds a million 8-byte pointers to them. A C++ <code>std::vector&#x3C;float></code> of a million floats takes 4 MB, because it holds the numbers themselves, side by side, four bytes each. That is also why NumPy and PyTorch exist: under their Python surface, a tensor is a C or C++ buffer of raw numbers.</p>\n<p>The difference is not an accident of implementation. Python decides where every object lives and when it is freed, and pays for that convenience in memory and speed. C++ hands both decisions to you. You choose whether a value lives on the <strong>stack</strong>, where it is free to create and disappears automatically, or on the <strong>heap</strong>, where it can be as large as you like and live as long as you need, but costs more to create and must be released. And you need to know exactly when each object's life ends, because using an object after that is one of the most common and most dangerous bugs in C++.</p>\n<p>This matters more in AI infrastructure than almost anywhere else. An inference engine is, to a large extent, a memory manager: vLLM's central idea, PagedAttention, is a way of allocating the KV cache, and PyTorch keeps its own allocator for GPU memory because asking the driver for memory on every operation would be far too slow. This chapter covers the CPU side of the same ideas: the stack, the heap, <code>new</code> and <code>delete</code>, the rules for when objects are created and destroyed, and how <code>sizeof</code>, alignment and padding decide how much memory a struct really takes. The lab watches objects being born and destroyed, measures struct layouts, and crashes a program three different ways on purpose.</p>","headings":[]},{"id":"where-python-keeps-your-data-and-where-c-does","title":"Where Python keeps your data, and where C++ does","html":"<h3 id=\"where-python-keeps-your-data-and-where-c-does\">Where Python keeps your data, and where C++ does</h3>\n<p>In CPython, every value is an object on the heap, and every variable is a reference to one. <code>x = 3.0</code> creates a float object and points <code>x</code> at it. The interpreter counts the references to each object and frees the object the moment the count reaches zero, and a garbage collector cleans up the cycles that counting misses. You never choose where anything lives, and you rarely think about when it is freed.</p>\n<p>In C++, a variable <em>is</em> its value. <code>double x = 3.0;</code> inside a function sets aside 8 bytes in that function's stack frame and puts the bits of 3.0 there. There is no separate object, no reference count, and no pointer to follow. Where a value lives depends on how you create it, and the language calls this its <strong>storage duration</strong>:</p>\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n<div class=\"table-wrap\"><table><thead><tr><th>How it is created</th><th>Where it lives</th><th>When it dies</th></tr></thead><tbody><tr><td>A local variable in a function</td><td>The stack</td><td>When its enclosing block ends</td></tr><tr><td>With <code>new</code>, or inside a container such as <code>std::vector</code></td><td>The heap</td><td>When <code>delete</code> runs, or the container frees it</td></tr><tr><td>A global, or a <code>static</code> variable</td><td>A fixed area set up when the program starts</td><td>After <code>main</code> returns</td></tr><tr><td><code>thread_local</code></td><td>Per thread</td><td>When its thread ends</td></tr></tbody></table></div>\n<p>Most real objects mix the first two. A <code>std::vector&#x3C;float></code> is a small object, three pointers or 24 bytes with GCC's and Clang's standard libraries on a 64-bit machine, that can live on the stack. Its elements live in a separate block on the heap, which the vector allocates, grows and frees for you:</p>\n<figure class=\"diagram\" data-text=\"main&#x27;s frame -> load_layer&#x27;s frame (calls); load_layer&#x27;s frame -> w: std::vector 24 bytes (local variable); w: std::vector 24 bytes -> 4,096 floats 16 KiB on the heap (data pointer)\"><div class=\"diagram-scroll\"><svg class=\"dg\" viewBox=\"0 0 720 205\" width=\"720\" role=\"img\" aria-label=\"A vector: a small object on the stack, its elements on the heap\"><defs><marker id=\"dg-arrow\" viewBox=\"0 0 10 10\" refX=\"9\" refY=\"5\" markerWidth=\"7\" markerHeight=\"7\" orient=\"auto-start-reverse\"><path class=\"dg-arrowhead\" d=\"M0,1 L9,5 L0,9 z\"></path></marker></defs><line class=\"dg-edge\" x1=\"80\" y1=\"69\" x2=\"80\" y2=\"133\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge\" x1=\"151\" y1=\"162\" x2=\"247\" y2=\"162\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge\" x1=\"391\" y1=\"162\" x2=\"567\" y2=\"162\" marker-end=\"url(#dg-arrow)\"></line><rect class=\"dg-node dg-box\" x=\"12\" y=\"18\" width=\"136\" height=\"48\" rx=\"8\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"80\" y=\"46.5\">main's frame</tspan></text><rect class=\"dg-node dg-box\" x=\"12\" y=\"138\" width=\"136\" height=\"48\" rx=\"8\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"80\" y=\"166.5\">load_layer's frame</tspan></text><rect class=\"dg-node dg-box\" x=\"252\" y=\"136\" width=\"136\" height=\"52\" rx=\"8\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"320\" y=\"158.5\">w: std::vector</tspan><tspan x=\"320\" y=\"174.5\">24 bytes</tspan></text><path class=\"dg-node dg-store\" d=\"M572,138 a68,7 0 0,1 136,0 v48 a68,7 0 0,1 -136,0 z\"></path><path class=\"dg-line\" d=\"M572,138 a68,7 0 0,0 136,0\"></path><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"640\" y=\"161.5\">4,096 floats</tspan><tspan x=\"640\" y=\"177.5\">16 KiB on the heap</tspan></text><rect class=\"dg-edge-bg\" x=\"59.5\" y=\"92\" width=\"41\" height=\"18\" rx=\"4\"></rect><text class=\"dg-edge-label\" x=\"80\" y=\"105\" text-anchor=\"middle\">calls</text><rect class=\"dg-edge-bg\" x=\"150.6\" y=\"114\" width=\"96.8\" height=\"18\" rx=\"4\"></rect><text class=\"dg-edge-label\" x=\"199\" y=\"127\" text-anchor=\"middle\">local variable</text><rect class=\"dg-edge-bg\" x=\"436.8\" y=\"153\" width=\"84.4\" height=\"18\" rx=\"4\"></rect><text class=\"dg-edge-label\" x=\"479\" y=\"166\" text-anchor=\"middle\">data pointer</text></svg></div><figcaption>A vector: a small object on the stack, its elements on the heap</figcaption></figure>\n<p>When <code>load_layer</code> returns, <code>w</code> is destroyed with its frame, and its destructor frees the heap block. Nothing waits for a garbage collector. This pattern, a small owner on the stack that frees a heap resource when it dies, is called RAII, and Chapter 6 is built around it.</p>","headings":[{"id":"where-python-keeps-your-data-and-where-c-does","title":"Where Python keeps your data, and where C++ does","depth":3}]},{"id":"the-stack-fast-automatic-and-small","title":"The stack: fast, automatic and small","html":"<h3 id=\"the-stack-fast-automatic-and-small\">The stack: fast, automatic and small</h3>\n<p>Every call to a function gets a <strong>stack frame</strong>: a block of memory holding the function's local variables, the arguments that did not fit in registers, and the address to return to. Frames are pushed and popped in strict order, last in, first out, which is where the name comes from. On x86-64 and ARM64 the stack grows downward, and creating a whole frame is usually one instruction that subtracts the frame's size from the stack pointer register. Destroying it is one instruction that adds the size back.</p>\n<p>That gives the stack three properties worth remembering:</p>\n<ul>\n<li><strong>It is nearly free.</strong> Allocating a local <code>float[64]</code> costs the same as allocating nothing, because the whole frame is reserved at once.</li>\n<li><strong>It is automatic.</strong> Locals disappear when their block ends, whatever path the code takes out of it: falling off the end, <code>return</code>, <code>break</code>, or an exception.</li>\n<li><strong>It is usually fast to touch.</strong> The same few kilobytes at the top of the stack are reused call after call, so they are almost always in the processor's cache.</li>\n</ul>\n<p>The catch is that a frame's size must be fixed when the function is compiled. <code>float scratch[n];</code> with an <code>n</code> known only at run time is not standard C++. GCC and Clang accept it as an extension called a variable-length array, and <code>-Wpedantic</code> warns about it. Data whose size you learn at run time, such as a tensor whose shape comes from a model file, belongs on the heap.</p>\n<h4 id=\"how-big-is-the-stack\">How big is the stack?</h4>\n<p>The other catch is that the stack is small, and it does not grow on demand beyond a fixed limit. The defaults are worth knowing because they differ between platforms:</p>\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n<div class=\"table-wrap\"><table><thead><tr><th>Platform</th><th>Main thread</th><th>Other threads</th></tr></thead><tbody><tr><td>Linux (glibc)</td><td>Usually 8 MB, from <code>ulimit -s</code></td><td>The same limit by default</td></tr><tr><td>macOS</td><td>8 MB</td><td>512 KB</td></tr><tr><td>Windows</td><td>1 MB</td><td>1 MB</td></tr></tbody></table></div>\n<p>Exceed the limit and the program touches a guard page below the stack, and the operating system kills it with a segmentation fault. There is no exception to catch and no friendly message. Two things cause this in practice. One is deep recursion, such as a recursive walk over a very deep tree. The other is a large local array: <code>float logits[32000 * 128];</code> is about 16 MB, twice the Linux default, and a program that declares it inside a function crashes as soon as it touches the array. The rule of thumb is to keep anything larger than a few kilobytes off the stack. Note the macOS row: code that runs happily on the main thread can crash in a worker thread, whose stack is sixteen times smaller.</p>","headings":[{"id":"the-stack-fast-automatic-and-small","title":"The stack: fast, automatic and small","depth":3},{"id":"how-big-is-the-stack","title":"How big is the stack?","depth":4}]},{"id":"the-heap-big-flexible-and-manual","title":"The heap: big, flexible and manual","html":"<h3 id=\"the-heap-big-flexible-and-manual\">The heap: big, flexible and manual</h3>\n<p>The heap is the large pool of memory that a program requests from the operating system as it needs it. A heap allocation can be any size, decided at run time, and it lives until you release it, whatever happens to the function that created it. That is exactly what tensors, KV caches and model weights need.</p>\n<h4 id=\"new-and-delete\">new and delete</h4>\n<p>The raw C++ tools are <code>new</code> and <code>delete</code>:</p>\n<pre class=\"shiki shiki-themes github-light github-dark\" style=\"background-color:#fff;--shiki-dark-bg:#24292e;color:#24292e;--shiki-dark:#e1e4e8\" tabindex=\"0\"><code><span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">float*</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> one </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> new</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> float</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">3.0</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">f</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">);</span><span style=\"color:#6A737D;--shiki-dark:#6A737D\">    // allocate one float on the heap and initialize it</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">delete</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> one;</span><span style=\"color:#6A737D;--shiki-dark:#6A737D\">                      // destroy it and give the memory back</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">float*</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> buf </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> new</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> float</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">[n];</span><span style=\"color:#6A737D;--shiki-dark:#6A737D\">       // allocate n floats; their values are indeterminate</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">float*</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> zeroed </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> new</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> float</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">[n]();</span><span style=\"color:#6A737D;--shiki-dark:#6A737D\">  // the () asks for every element to be zero</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">delete[]</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> buf;</span><span style=\"color:#6A737D;--shiki-dark:#6A737D\">                    // an array from new[] must be released with delete[]</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">delete[]</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> zeroed;</span></span></code></pre>\n<p><code>new</code> does two things: it asks the allocator for memory, and then runs the object's constructor in it. <code>delete</code> does the reverse: it runs the destructor and returns the memory. If the allocator cannot find memory, <code>new</code> throws <code>std::bad_alloc</code> rather than returning a null pointer, unlike C's <code>malloc</code>. On Linux, though, a large allocation often succeeds even when memory is short, because the kernel assigns physical pages only when the program first writes to them, so running out tends to show up later, as the out-of-memory killer ending the process.</p>\n<p>Manual <code>new</code> and <code>delete</code> give you three ways to get it wrong, and each is undefined behavior or a leak:</p>\n<ol>\n<li><strong>Forget the <code>delete</code></strong>, and the memory is never returned. This is a <strong>leak</strong>. In a server that runs for weeks, a few leaked kilobytes per request add up to an out-of-memory crash.</li>\n<li><strong>Use the memory after <code>delete</code></strong>, and you read or write a block that the allocator may already have handed to someone else. This is a <strong>use-after-free</strong>.</li>\n<li><strong><code>delete</code> the same pointer twice</strong>, or release <code>new[]</code> memory with plain <code>delete</code>, and you corrupt the allocator's own records. This is a <strong>double free</strong> or a mismatched delete.</li>\n</ol>\n<p>The first is easy to commit, because every path out of a function must reach the <code>delete</code>:</p>\n<pre class=\"shiki shiki-themes github-light github-dark\" style=\"background-color:#fff;--shiki-dark-bg:#24292e;color:#24292e;--shiki-dark:#e1e4e8\" tabindex=\"0\"><code><span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">float</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\"> score</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">int</span><span style=\"color:#E36209;--shiki-dark:#FFAB70\"> n_tokens</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">) {</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">  float*</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> scratch </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> new</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> float</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">[</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">1024</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> *</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 1024</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">];</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">  if</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> (n_tokens </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">==</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 0</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">) </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">return</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 0.0</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">f</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">;</span><span style=\"color:#6A737D;--shiki-dark:#6A737D\">   // leaks 4 MiB: this path never reaches delete[]</span></span>\n<span class=\"line\"><span style=\"color:#6A737D;--shiki-dark:#6A737D\">  // ... use scratch ...</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">  delete[]</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> scratch;</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">  return</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> sum;</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">}</span></span></code></pre>\n<p>An exception thrown in the middle leaks it too. The fix is not to be more careful. It is to stop writing <code>new</code> and <code>delete</code> by hand. Use <code>std::vector&#x3C;float> scratch(1024 * 1024);</code> and the vector frees its memory however the function exits. When you need a single heap object, <code>std::make_unique</code> (Chapter 8) does the same. Modern C++ code almost never contains a bare <code>delete</code>, and when you see one in a code review, it deserves a second look. You still need to understand <code>new</code> and <code>delete</code>, because containers and smart pointers are built on them, and because older code and C libraries are full of them.</p>\n<h4 id=\"what-an-allocation-costs\">What an allocation costs</h4>\n<p>A stack allocation is a subtraction. A heap allocation is a function call into the allocator, which has to find a free block of the right size, record that it is in use, and, in a multithreaded program, avoid colliding with other threads doing the same. For a large request it may also have to ask the operating system for fresh pages; glibc's <code>malloc</code>, for example, serves requests of 128 KiB and more with a separate <code>mmap</code> call by default, although it adjusts that threshold as the program runs. None of this is slow for one allocation, but an allocation inside a loop that runs a million times is a common reason C++ code is slower than expected.</p>\n<p>Even Python cares about this: CPython has its own small-object allocator, pymalloc, for objects of 512 bytes or less, because calling <code>malloc</code> for every integer and float would be too slow. AI systems take the idea further:</p>\n<ul>\n<li>PyTorch does not call <code>cudaMalloc</code> and <code>cudaFree</code> for every tensor. Its caching allocator keeps freed GPU blocks and reuses them, because those driver calls are expensive and <code>cudaFree</code> waits for the GPU to finish its work. The <a href=\"/learn/pytorch-execution#the-cuda-caching-allocator\">caching allocator section</a> of the infra handbook shows how it works.</li>\n<li>vLLM allocates the KV cache once, at startup, as one large pool of fixed-size blocks sized by <code>gpu_memory_utilization</code>, and then hands blocks to requests from that pool. The <a href=\"/learn/mini-inference-engine#the-paged-kv-cache\">paged KV cache section</a> builds the same thing in Python.</li>\n</ul>\n<p>The pattern in both is the same, and you will write it yourself in the capstone: allocate big blocks rarely, then reuse them.</p>","headings":[{"id":"the-heap-big-flexible-and-manual","title":"The heap: big, flexible and manual","depth":3},{"id":"new-and-delete","title":"new and delete","depth":4},{"id":"what-an-allocation-costs","title":"What an allocation costs","depth":4}]},{"id":"lifetime-when-objects-are-born-and-die","title":"Lifetime: when objects are born and die","html":"<h3 id=\"lifetime-when-objects-are-born-and-die\">Lifetime: when objects are born and die</h3>\n<p>Every C++ object has a <strong>lifetime</strong>: it begins when the object's construction finishes and ends when its destructor starts. Outside its lifetime, the memory may still be there, but there is no object in it, and using it is undefined behavior. Python never lets you see this, because an object lives as long as anyone holds a reference to it. In C++ the rules are fixed and predictable, which is what makes C++ resource management work:</p>\n<ul>\n<li>A local object is destroyed when its enclosing block ends, and locals in the same block are destroyed in the <strong>reverse order</strong> of their construction. Reverse order is what you want: an object created later may depend on one created earlier, so it must go first.</li>\n<li>A <strong>temporary</strong>, such as the result of a function call that is never given a name, is destroyed at the end of the full expression that created it, which is usually the end of the statement.</li>\n<li>An object created with <code>new</code> lives until <code>delete</code>, regardless of scopes.</li>\n<li>A global is constructed before <code>main</code> starts and destroyed after <code>main</code> returns. A <code>static</code> variable inside a function is constructed the first time control reaches it, safely even if several threads arrive at once, and is also destroyed after <code>main</code> returns. Statics are destroyed in the reverse order of their construction.</li>\n</ul>\n<p>The lab's <code>lifetime.cpp</code> makes this visible with a struct that prints a line in its constructor and its destructor:</p>\n<pre class=\"shiki shiki-themes github-light github-dark\" style=\"background-color:#fff;--shiki-dark-bg:#24292e;color:#24292e;--shiki-dark:#e1e4e8\" tabindex=\"0\"><code><span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">struct</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\"> Tracer</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> {</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">  const</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> char*</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> name;</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">  explicit</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\"> Tracer</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">const</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> char*</span><span style=\"color:#E36209;--shiki-dark:#FFAB70\"> n</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">) : </span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\">name</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(n) { </span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\">std</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">::</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\">printf</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"  construct </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">%s\\n</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, name); }</span></span>\n<span class=\"line\"><span style=\"color:#6F42C1;--shiki-dark:#B392F0\">  ~Tracer</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">() { </span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\">std</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">::</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\">printf</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"  destroy   </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">%s\\n</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, name); }</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">};</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">void</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\"> scopes</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">() {</span></span>\n<span class=\"line\"><span style=\"color:#6F42C1;--shiki-dark:#B392F0\">  Tracer</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\"> a</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"a\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">);</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">  {</span></span>\n<span class=\"line\"><span style=\"color:#6F42C1;--shiki-dark:#B392F0\">    Tracer</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\"> b</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"b\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">);</span></span>\n<span class=\"line\"><span style=\"color:#6F42C1;--shiki-dark:#B392F0\">    Tracer</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\"> c</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"c\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">);</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">  }</span><span style=\"color:#6A737D;--shiki-dark:#6A737D\">               // c is destroyed, then b</span></span>\n<span class=\"line\"><span style=\"color:#6F42C1;--shiki-dark:#B392F0\">  Tracer</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\"> d</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"d\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">);</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">}</span><span style=\"color:#6A737D;--shiki-dark:#6A737D\">                 // d is destroyed, then a</span></span></code></pre>\n<p>The order is not a quirk of one compiler. The standard requires it, so every conforming compiler prints <code>a</code>, <code>b</code>, <code>c</code> constructed, then <code>c</code> and <code>b</code> destroyed, then <code>d</code> constructed, and finally <code>d</code> and <code>a</code> destroyed. This determinism is a real advantage over Python. A file, a lock or a block of GPU memory owned by a C++ object is released at a known point in the code, while Python releases it whenever the last reference disappears, which in practice is often but not always immediately, and is why Python code needs <code>with</code> statements for files.</p>\n<h4 id=\"dangling-a-pointer-that-outlives-its-object\">Dangling: a pointer that outlives its object</h4>\n<p>A <strong>dangling</strong> pointer or reference refers to an object whose lifetime has ended. It still holds an address, and reading through it may even seem to work, because the old bytes are often still there. That is what makes it dangerous: the bug shows up later, somewhere else, as wrong data.</p>\n<p>The textbook case is returning a reference to a local variable:</p>\n<pre class=\"shiki shiki-themes github-light github-dark\" style=\"background-color:#fff;--shiki-dark-bg:#24292e;color:#24292e;--shiki-dark:#e1e4e8\" tabindex=\"0\"><code><span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">const</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\"> std</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">::</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\">vector</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">&#x3C;</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">float</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">></span><span style=\"color:#D73A49;--shiki-dark:#F97583\">&#x26;</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\"> make_logits</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">() {</span></span>\n<span class=\"line\"><span style=\"color:#6F42C1;--shiki-dark:#B392F0\">  std</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">::</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\">vector</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">&#x3C;</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">float</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">> </span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\">logits</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">32000</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">);</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">  return</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> logits;</span><span style=\"color:#6A737D;--shiki-dark:#6A737D\">   // logits is destroyed here, so the caller gets a reference to a dead object</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">}</span></span></code></pre>\n<p>Compilers catch this simple form. GCC and Clang both warn by default that the function returns a reference to a local, and the fix is to return the vector by value, which C++ makes cheap, as Chapter 7 explains.</p>\n<p>The case that bites real systems is subtler, because nothing is returned and no compiler warns. A <code>std::vector</code> keeps its elements in one heap block with some spare room, its <strong>capacity</strong>. When you add an element and the spare room is used up, the vector allocates a larger block, moves the elements across, and frees the old block. Any pointer or reference into the old block now dangles:</p>\n<pre class=\"shiki shiki-themes github-light github-dark\" style=\"background-color:#fff;--shiki-dark-bg:#24292e;color:#24292e;--shiki-dark:#e1e4e8\" tabindex=\"0\"><code><span class=\"line\"><span style=\"color:#6F42C1;--shiki-dark:#B392F0\">std</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">::vector</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">&#x3C;float></span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> cache;</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">cache.</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\">push_back</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">1.0</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">f</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">);</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">float*</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> first </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> cache.</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\">data</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">();</span><span style=\"color:#6A737D;--shiki-dark:#6A737D\">          // points into the current block</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">for</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> (</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">int</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> i </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 0</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">; i </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">&#x3C;</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 1000</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">; </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">++</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">i) cache.</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\">push_back</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(i);</span><span style=\"color:#6A737D;--shiki-dark:#6A737D\">   // the vector reallocates, several times</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">float</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> x </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> *</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">first;</span><span style=\"color:#6A737D;--shiki-dark:#6A737D\">                     // undefined behavior: that block was freed</span></span></code></pre>\n<p>Imagine this in an inference engine: one component keeps a pointer to a sequence's KV entries while another appends a new token, and attention silently reads freed memory. It produces plausible numbers, not a crash, at least until the allocator reuses the block. Two habits prevent it. Keep indexes rather than pointers into a container that can grow, or reserve the full size up front with <code>cache.reserve(n)</code>, which allocates once so that later <code>push_back</code> calls up to that size never move the elements. Chapter 5 shows how AddressSanitizer reports this bug the moment it happens, and the lab gives you a preview.</p>","headings":[{"id":"lifetime-when-objects-are-born-and-die","title":"Lifetime: when objects are born and die","depth":3},{"id":"dangling-a-pointer-that-outlives-its-object","title":"Dangling: a pointer that outlives its object","depth":4}]},{"id":"sizeof-alignment-and-padding","title":"sizeof, alignment and padding","html":"<h3 id=\"sizeof-alignment-and-padding\">sizeof, alignment and padding</h3>\n<p><code>sizeof(T)</code> is the number of bytes an object of type <code>T</code> occupies, computed while compiling. For the basic types on mainstream 64-bit platforms the answers are the ones you would guess: 1 for <code>bool</code> and <code>char</code>, 4 for <code>int</code> and <code>float</code>, 8 for <code>double</code>, <code>std::int64_t</code> and any pointer. For a struct, though, the answer is often larger than the sum of its members, and the reason is <strong>alignment</strong>.</p>\n<p>A processor reads memory most efficiently when a value starts at an address that is a multiple of its size, and some instructions require it. So every type has an alignment, given by <code>alignof(T)</code>: an 8-byte <code>double</code> must start at an address divisible by 8. To keep every member of a struct aligned, the compiler applies three rules:</p>\n<ol>\n<li>Members are laid out in the order you declare them.</li>\n<li>Each member starts at the next offset that is a multiple of its alignment, and the compiler inserts unused <strong>padding</strong> bytes to get there.</li>\n<li>The struct's alignment is its largest member's alignment, and its size is rounded up to a multiple of that, so that the next element of an array is aligned too.</li>\n</ol>\n<p>Here is per-block bookkeeping for a KV cache, written in the order someone thought of the fields:</p>\n<pre class=\"shiki shiki-themes github-light github-dark\" style=\"background-color:#fff;--shiki-dark-bg:#24292e;color:#24292e;--shiki-dark:#e1e4e8\" tabindex=\"0\"><code><span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">struct</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\"> BlockInfo</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> {</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">  bool</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> in_use;</span><span style=\"color:#6A737D;--shiki-dark:#6A737D\">             // offset 0, then 7 bytes of padding</span></span>\n<span class=\"line\"><span style=\"color:#6F42C1;--shiki-dark:#B392F0\">  std</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">::</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">int64_t</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> seq_id;</span><span style=\"color:#6A737D;--shiki-dark:#6A737D\">     // offset 8</span></span>\n<span class=\"line\"><span style=\"color:#6F42C1;--shiki-dark:#B392F0\">  std</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">::</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">int32_t</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> ref_count;</span><span style=\"color:#6A737D;--shiki-dark:#6A737D\">  // offset 16</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">  bool</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> on_gpu;</span><span style=\"color:#6A737D;--shiki-dark:#6A737D\">             // offset 20, then 3 bytes of padding</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">  double</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> last_used;</span><span style=\"color:#6A737D;--shiki-dark:#6A737D\">        // offset 24</span></span>\n<span class=\"line\"><span style=\"color:#6F42C1;--shiki-dark:#B392F0\">  std</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">::</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">int32_t</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> num_tokens;</span><span style=\"color:#6A737D;--shiki-dark:#6A737D\"> // offset 32, then 4 bytes of padding at the end</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">};</span><span style=\"color:#6A737D;--shiki-dark:#6A737D\">                         // sizeof 40, of which 14 bytes are padding</span></span></code></pre>\n<p>The members hold 26 bytes of data, but the struct takes 40. Sort the same members from largest alignment to smallest, <code>seq_id</code>, <code>last_used</code>, <code>ref_count</code>, <code>num_tokens</code>, <code>in_use</code>, <code>on_gpu</code>, and the only padding left is 6 bytes at the end: <code>sizeof</code> drops to 32, 20% smaller, with no change in meaning. For one struct this is irrelevant. For a table with an entry per KV block, per token or per request, multiplied by millions, it is memory and cache space for free.</p>\n<div class=\"widget-slot\" data-widget=\"struct-layout\"></div>\n<p>Three related tools are worth knowing now, and later chapters use them:</p>\n<ul>\n<li><code>offsetof(T, member)</code>, from <code>&#x3C;cstddef></code>, gives a member's offset in bytes. Binary file formats and GPU code that share a struct between two programs check it with <code>static_assert</code>, because both sides must agree on every byte.</li>\n<li><code>alignas(64)</code> raises a type's alignment. Two threads that update counters sitting in the same 64-byte cache line slow each other down, an effect called false sharing, and padding each counter to its own line is the fix. Chapter 20 covers it, and Chapter 13 explains cache lines.</li>\n<li>Packed layouts, written <code>#pragma pack</code> or <code>__attribute__((packed))</code>, remove padding entirely, at the price of unaligned access. They appear in code that reads file headers byte for byte. llama.cpp's quantized blocks avoid the need by choosing members that pack naturally: <code>block_q4_0</code> is a 2-byte scale followed by 16 bytes of 4-bit weights, 18 bytes with no padding, because a 2-byte alignment fits evenly into 18.</li>\n</ul>\n<p><code>sizeof</code> also explains the standard library's own types. <code>sizeof(std::vector&#x3C;float>)</code> is 24 on 64-bit Linux and macOS no matter how many elements the vector holds, because the elements are on the heap. <code>sizeof(std::string)</code> is 32 with GCC's libstdc++ and 24 with LLVM's libc++, which Apple clang uses: both store short strings inside the string object itself, a trick called the small string optimization, and they reserve different amounts of room for it. A single empty struct, finally, has a <code>sizeof</code> of 1, not 0, because every object must have its own address.</p>","headings":[{"id":"sizeof-alignment-and-padding","title":"sizeof, alignment and padding","depth":3}]},{"id":"lab-lifetimes-layouts-and-three-crashes","title":"Lab: lifetimes, layouts and three crashes","html":"<h3 id=\"lab-lifetimes-layouts-and-three-crashes\">Lab: lifetimes, layouts and three crashes</h3>\n<p>The lab is three small programs: <a href=\"/labs/cpp/cpp-memory/lifetime.cpp\"><code>lifetime.cpp</code></a>, <a href=\"/labs/cpp/cpp-memory/layout.cpp\"><code>layout.cpp</code></a> and <a href=\"/labs/cpp/cpp-memory/memory_demo.cpp\"><code>memory_demo.cpp</code></a>, with a <a href=\"/labs/cpp/cpp-memory/CMakeLists.txt\"><code>CMakeLists.txt</code></a> that builds all three. Download the four files into one folder. It needs only a C++20 compiler and CMake. Build in Debug, so that the optimizer does not delete the work the crash demonstrations depend on:</p>\n<pre class=\"shiki shiki-themes github-light github-dark\" style=\"background-color:#fff;--shiki-dark-bg:#24292e;color:#24292e;--shiki-dark:#e1e4e8\" tabindex=\"0\"><code><span class=\"line\"><span style=\"color:#6F42C1;--shiki-dark:#B392F0\">cmake</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> -S</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> .</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> -B</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> build</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> -DCMAKE_BUILD_TYPE=Debug</span></span>\n<span class=\"line\"><span style=\"color:#6F42C1;--shiki-dark:#B392F0\">cmake</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> --build</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> build</span></span></code></pre>\n<h4 id=\"part-1-predict-the-order\">Part 1: predict the order</h4>\n<p>Read <code>lifetime.cpp</code> before running it. It has a global <code>Tracer</code>, the <code>scopes</code> function above, a temporary, an object created with <code>new</code>, and a function-local <code>static</code> that is reached twice. Write down the complete order of <code>construct</code> and <code>destroy</code> lines you expect, then run <code>./build/lifetime</code> and compare. Pay attention to three lines in particular: <code>construct global</code> comes before <code>main starts</code>; <code>destroy temp</code> comes after the line that printed the temporary's name, because the temporary lives until the end of the whole <code>printf</code> statement; and the function-local static is constructed once, on the first call, and destroyed after <code>main returns</code>, before the global.</p>\n<h4 id=\"part-2-measure-the-layouts\">Part 2: measure the layouts</h4>\n<p>Run <code>./build/layout</code>. It prints <code>sizeof</code> and <code>alignof</code> for the built-in types, for <code>std::vector&#x3C;float></code> and <code>std::string</code>, and for both versions of <code>BlockInfo</code> with every member's offset. Check the offsets against the comments in the <code>BlockInfo</code> listing above. The file also contains five <code>static_assert</code> checks of these sizes, so if it compiled at all, they held on your machine.</p>\n<p>Two lines in the output depend on your standard library. <code>sizeof(std::string)</code> should be 32 with GCC and 24 with Apple clang, and the program prints how many characters an empty string can hold before it needs the heap, which differs for the same reason. If you have both compilers, run it with each. <code>long double</code> is another line that differs: it is 16 bytes on x86-64 Linux but the same as <code>double</code> on ARM64 macOS.</p>\n<h4 id=\"part-3-three-crashes-on-purpose\">Part 3: three crashes on purpose</h4>\n<p><code>memory_demo</code> takes one mode at a time:</p>\n<pre class=\"shiki shiki-themes github-light github-dark\" style=\"background-color:#fff;--shiki-dark-bg:#24292e;color:#24292e;--shiki-dark:#e1e4e8\" tabindex=\"0\"><code><span class=\"line\"><span style=\"color:#6F42C1;--shiki-dark:#B392F0\">./build/memory_demo</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> heap-big</span><span style=\"color:#6A737D;--shiki-dark:#6A737D\">       # 16 MiB of floats in a std::vector: works</span></span>\n<span class=\"line\"><span style=\"color:#6F42C1;--shiki-dark:#B392F0\">./build/memory_demo</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> stack-big</span><span style=\"color:#6A737D;--shiki-dark:#6A737D\">      # the same 16 MiB as a local array: crashes</span></span>\n<span class=\"line\"><span style=\"color:#6F42C1;--shiki-dark:#B392F0\">./build/memory_demo</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> leak</span><span style=\"color:#6A737D;--shiki-dark:#6A737D\">           # loses 4 MiB on each of ten early returns</span></span>\n<span class=\"line\"><span style=\"color:#6F42C1;--shiki-dark:#B392F0\">./build/memory_demo</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> dangling</span><span style=\"color:#6A737D;--shiki-dark:#6A737D\">       # keeps a pointer into a vector that grows</span></span>\n<span class=\"line\"><span style=\"color:#6F42C1;--shiki-dark:#B392F0\">./build/memory_demo</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> dangling-read</span><span style=\"color:#6A737D;--shiki-dark:#6A737D\">  # and then reads through it</span></span></code></pre>\n<p>The first two do the same work, and only the location of the buffer differs. <code>stack-big</code> should die with a segmentation fault and no other explanation, because a 16 MiB frame does not fit in an 8 MB stack. <code>leak</code> runs to the end without complaint, which is the whole problem with leaks. <code>dangling</code> prints the vector's data address before and after it grows, so you can see that the elements moved, and <code>dangling-read</code> then reads through the stale pointer. It will probably print a number, perhaps even the right one, because the freed block has not been reused yet. That is undefined behavior working exactly as described: no crash, no warning, possibly the wrong answer.</p>\n<p>Now rebuild with the sanitizers and run the same modes again:</p>\n<pre class=\"shiki shiki-themes github-light github-dark\" style=\"background-color:#fff;--shiki-dark-bg:#24292e;color:#24292e;--shiki-dark:#e1e4e8\" tabindex=\"0\"><code><span class=\"line\"><span style=\"color:#6F42C1;--shiki-dark:#B392F0\">cmake</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> -S</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> .</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> -B</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> build-san</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> -DCMAKE_BUILD_TYPE=Debug</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> -DLAB_SANITIZE=ON</span></span>\n<span class=\"line\"><span style=\"color:#6F42C1;--shiki-dark:#B392F0\">cmake</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> --build</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> build-san</span></span>\n<span class=\"line\"><span style=\"color:#6F42C1;--shiki-dark:#B392F0\">./build-san/memory_demo</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> stack-big</span></span>\n<span class=\"line\"><span style=\"color:#6F42C1;--shiki-dark:#B392F0\">./build-san/memory_demo</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> dangling-read</span></span>\n<span class=\"line\"><span style=\"color:#6F42C1;--shiki-dark:#B392F0\">./build-san/memory_demo</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> leak</span></span></code></pre>\n<p>AddressSanitizer turns the silent bugs into reports. <code>stack-big</code> should now stop with an AddressSanitizer report and a stack trace, usually headed <code>ERROR: AddressSanitizer: stack-overflow</code>, although some platforms report it as a plain <code>SEGV</code> instead, and <code>dangling-read</code> should stop at the bad read with <code>ERROR: AddressSanitizer: heap-use-after-free</code>, followed by three stack traces: where the bad read happened, where the memory was freed (inside <code>push_back</code>, as the vector grew), and where it was originally allocated. That is often enough to find the bug without a debugger. For the leak, it depends on the platform. On Linux, AddressSanitizer includes LeakSanitizer, which reports <code>detected memory leaks</code> when the program exits, with the stack of each leaked allocation. Apple's clang does not support LeakSanitizer, so on a Mac use the system's <code>leaks</code> tool on the normal build instead: <code>leaks -atExit -- ./build/memory_demo leak</code>.</p>\n<h4 id=\"stretch-goals\">Stretch goals</h4>\n<ul>\n<li>In <code>dangling</code>, call <code>cache.reserve(1001);</code> right after creating the vector and run <code>dangling-read</code> under the sanitizer again. The data address should no longer change, and the report should disappear.</li>\n<li>Fix <code>score</code> so that it cannot leak, by replacing the <code>new[]</code> with a <code>std::vector&#x3C;float></code>, and confirm that the leak report is gone.</li>\n<li>Run <code>ulimit -s</code> in your shell to see the stack limit, in kilobytes, and <code>ulimit -Hs</code> to see the hard limit, the highest value you are allowed to set. On Linux, <code>ulimit -s 32768</code> raises the limit to 32 MB for programs started from that shell, after which <code>stack-big</code> should succeed. Try the same on macOS and note what happens.</li>\n</ul>","headings":[{"id":"lab-lifetimes-layouts-and-three-crashes","title":"Lab: lifetimes, layouts and three crashes","depth":3},{"id":"part-1-predict-the-order","title":"Part 1: predict the order","depth":4},{"id":"part-2-measure-the-layouts","title":"Part 2: measure the layouts","depth":4},{"id":"part-3-three-crashes-on-purpose","title":"Part 3: three crashes on purpose","depth":4},{"id":"stretch-goals","title":"Stretch goals","depth":4}]},{"id":"check-yourself","title":"Check yourself","html":"<h3 id=\"check-yourself\">Check yourself</h3>\n<details>\n<summary>Why is creating a local array on the stack cheaper than allocating the same array with new?</summary>\n<p>A function's whole stack frame, including every local array, is reserved at once by moving the stack pointer, usually with one instruction, and released the same way on return. <code>new</code> calls the allocator, which must find a free block of the right size, record that it is in use and coordinate with other threads, and it must be matched by a <code>delete</code> that does the bookkeeping in reverse.</p>\n</details>\n<details>\n<summary>A function declares float logits[32000 * 128]; as a local variable and crashes with a segmentation fault the first time it runs. Why, and what is the fix?</summary>\n<p>The array is 4,096,000 floats, about 16 MB, which is larger than the usual 8 MB stack on Linux and macOS, so touching it runs past the end of the stack into the guard page. The fix is to put it on the heap, for example <code>std::vector&#x3C;float> logits(32000 * 128);</code>, where the vector object on the stack is only 24 bytes.</p>\n</details>\n<details>\n<summary>In what order are a, b and c destroyed if they are declared in that order in one block, and why that order?</summary>\n<p>In reverse: <code>c</code>, then <code>b</code>, then <code>a</code>. Objects declared later may depend on objects declared earlier, for example a lock guard that refers to a mutex declared above it, so destroying them in reverse order guarantees that nothing is destroyed while a later object might still use it.</p>\n</details>\n<details>\n<summary>Code keeps a float* to an element of a std::vector&#x3C;float>, and after more push_back calls the values read through it are wrong. What happened?</summary>\n<p>When a <code>push_back</code> exceeds the vector's capacity, the vector allocates a larger block, moves the elements to it and frees the old block. The pointer still holds the old address, so it dangles, and reading through it is a use-after-free. Store an index instead of a pointer, or call <code>reserve</code> with the final size first so that the vector never reallocates.</p>\n</details>\n<details>\n<summary>A struct holds a bool, a double and a bool, in that order. What is its sizeof on a 64-bit platform, and how can you shrink it?</summary>\n<p>24 bytes: the first <code>bool</code> is at offset 0, followed by 7 bytes of padding so the <code>double</code> can start at offset 8; the second <code>bool</code> is at offset 16, and the size is rounded up to a multiple of 8, giving 24. Putting the <code>double</code> first and the two <code>bool</code>s after it makes the layout 8 + 1 + 1 bytes, rounded up to 16.</p>\n</details>\n<details>\n<summary>Why do PyTorch and vLLM manage GPU memory with their own allocators or pools instead of allocating for each tensor?</summary>\n<p>Asking the driver for memory with <code>cudaMalloc</code> and returning it with <code>cudaFree</code> is expensive, and <code>cudaFree</code> waits for the GPU to finish its work. PyTorch's caching allocator keeps freed blocks and reuses them, and vLLM allocates its whole KV cache as a pool of fixed-size blocks at startup and hands them out to requests. Both follow the same rule as fast CPU code: allocate large blocks rarely and reuse them.</p>\n</details>\n<h3 id=\"further-reading\">Further reading</h3>\n<ul>\n<li><a href=\"https://en.cppreference.com/w/cpp/language/storage_duration\" target=\"_blank\" rel=\"noopener noreferrer\">cppreference: storage duration</a> and <a href=\"https://en.cppreference.com/w/cpp/language/lifetime\" target=\"_blank\" rel=\"noopener noreferrer\">cppreference: lifetime</a> give the precise rules for when objects are created and destroyed, including temporaries.</li>\n<li><a href=\"https://en.cppreference.com/w/cpp/language/new\" target=\"_blank\" rel=\"noopener noreferrer\">cppreference: new expression</a> and <a href=\"https://en.cppreference.com/w/cpp/language/delete\" target=\"_blank\" rel=\"noopener noreferrer\">delete expression</a> describe what <code>new</code> and <code>delete</code> do, including the array forms.</li>\n<li><a href=\"https://en.cppreference.com/w/cpp/language/object#Alignment\" target=\"_blank\" rel=\"noopener noreferrer\">cppreference: object, alignment</a> explains alignment and padding, and <a href=\"https://en.cppreference.com/w/cpp/language/alignas\" target=\"_blank\" rel=\"noopener noreferrer\"><code>alignas</code></a> shows how to raise it.</li>\n<li><a href=\"http://www.catb.org/esr/structure-packing/\" target=\"_blank\" rel=\"noopener noreferrer\">The Lost Art of Structure Packing</a>, by Eric S. Raymond, is a long, practical guide to padding and how to reorder members to remove it.</li>\n<li><a href=\"https://clang.llvm.org/docs/AddressSanitizer.html\" target=\"_blank\" rel=\"noopener noreferrer\">AddressSanitizer in the Clang documentation</a> lists the bugs it finds and the platforms where leak detection works.</li>\n<li><a href=\"https://developer.apple.com/library/archive/documentation/Cocoa/Conceptual/Multithreading/CreatingThreads/CreatingThreads.html\" target=\"_blank\" rel=\"noopener noreferrer\">Apple's Threading Programming Guide</a> documents the 8 MB main-thread and 512 KB secondary-thread stack sizes on macOS.</li>\n<li><a href=\"https://zdevito.github.io/2022/08/04/cuda-caching-allocator.html\" target=\"_blank\" rel=\"noopener noreferrer\">A guide to PyTorch's CUDA caching allocator</a>, by Zachary DeVito, shows the same \"allocate rarely, reuse often\" idea applied to GPU memory.</li>\n</ul>","headings":[{"id":"check-yourself","title":"Check yourself","depth":3},{"id":"further-reading","title":"Further reading","depth":3}]}]}