{"id":"other-compression","title":"Pruning and distillation","sections":[{"id":"overview","title":"Overview","html":"<p>Quantization keeps every weight and stores it in fewer bits. The two other ways to make a model cheaper remove work instead. Pruning sets weights to zero, either anywhere or in a pattern the hardware can skip. Distillation trains a smaller model to imitate a larger one. Both are older than LLM quantization and both appear in interviews, but in serving practice they play a smaller role, for reasons this short chapter explains with measurements: a small model pruned on a Mac, and 2:4 sparse matmuls timed on an H100 and an L4.</p>\n<p>Before this chapter, read <a href=\"/learn/ptq-algorithms\">Post-training quantization algorithms</a>; the pruning methods here use the same calibration statistics.</p>","headings":[]},{"id":"unstructured-and-structured-pruning","title":"Unstructured and structured pruning","html":"<h3 id=\"unstructured-and-structured-pruning\">Unstructured and structured pruning</h3>\n<p>Unstructured pruning removes individual weights wherever they matter least, so it loses the least accuracy for a given sparsity, but the zeros are scattered, and a GPU's dense tensor cores multiply them anyway. Storing a scattered sparse matrix in the common formats needs an index per nonzero, so at 50% sparsity it saves no memory, and common GPU libraries gain no time from it. Structured pruning removes whole units: attention heads, MLP neurons, entire layers. The result is a smaller dense model that runs faster on any hardware, but removing whole units hurts more, and the model usually needs further training to recover.</p>\n<p>Semi-structured sparsity sits between the two. NVIDIA's Ampere and later GPUs have sparse tensor cores that accept a weight matrix in which every group of four consecutive values holds at least two zeros, the 2:4 pattern. The matrix is stored as its nonzero half plus a 2-bit index for each kept value, and NVIDIA states that the sparse tensor cores deliver twice the math throughput of dense ones; its paper measured nearly 2 times the speed on large matrix multiplications. The pattern is fixed at 50% sparsity, and since every group of four must lose two values, even a group of four important weights loses two.</p>","headings":[{"id":"unstructured-and-structured-pruning","title":"Unstructured and structured pruning","depth":3}]},{"id":"choosing-what-to-prune","title":"Choosing what to prune","html":"<h3 id=\"choosing-what-to-prune\">Choosing what to prune</h3>\n<p>Magnitude pruning, the classic method, removes the weights with the smallest absolute values. It ignores the inputs, which chapters 4.2 and 4.3 showed matter: a small weight on a channel with large activations can contribute more to the output than a large weight on a quiet channel. Two methods from 2023 fixed this for LLMs without retraining. SparseGPT, by Frantar and Alistarh, two of the GPTQ authors, uses the same Hessian-based error compensation as GPTQ: it removes weights and updates the remaining ones to preserve the layer's output, and it pruned OPT-175B and BLOOM-176B to 60% unstructured sparsity in under 4.5 hours with a negligible rise in perplexity. Wanda, by Sun and others, is simpler still: it scores each weight by its magnitude times the norm of its input channel over calibration data, |W_ij| × ||X_j||, removes the lowest-scoring weights in each output row, and updates nothing.</p>\n<div class=\"widget-slot\" data-widget=\"sparsity-pattern\"></div>\n<p>With magnitude scoring, notice how many weights on the three loud input channels are removed, and the output error that causes. Switch to Wanda's score: those weights survive and the error falls, without changing any kept weight. Then switch between the free pattern and 2:4 to see what the hardware's constraint costs.</p>","headings":[{"id":"choosing-what-to-prune","title":"Choosing what to prune","depth":3}]},{"id":"measured-half-the-weights-of-a-small-model","title":"Measured: half the weights of a small model","html":"<h3 id=\"measured-half-the-weights-of-a-small-model\">Measured: half the weights of a small model</h3>\n<p>The lab pruned every linear layer of SmolLM2-135M to 50% sparsity, scoring weights by magnitude and by Wanda's score with input norms from 16,384 calibration tokens, and measured WikiText-2 perplexity, which is 17.10 for the dense model:</p>\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n<div class=\"table-wrap\"><table><thead><tr><th>Pruning, 50% sparsity</th><th>Perplexity</th><th>Change</th></tr></thead><tbody><tr><td>Magnitude, unstructured</td><td>466.0</td><td>+2,625%</td></tr><tr><td>Wanda, unstructured</td><td>36.3</td><td>+112%</td></tr><tr><td>Magnitude, 2:4</td><td>1,477.2</td><td>+8,537%</td></tr><tr><td>Wanda, 2:4</td><td>104.6</td><td>+512%</td></tr><tr><td>For comparison: INT4 group 128, GPTQ with act-order (chapter 4.3)</td><td>20.1</td><td>+17%</td></tr></tbody></table></div>\n<p>Using the activations makes the difference between a broken model and a damaged one: Wanda's score cut the perplexity by a factor of 13 to 14 over magnitude pruning, in both patterns. The 2:4 constraint costs a further factor of about 3. And no pruning result comes near INT4 quantization, which stores about 4.2 bits per weight against 2:4 sparse BF16's 9 bits. Small models suffer more from pruning, as from quantization; the Wanda paper reports much smaller losses for 7B to 65B models at 50% unstructured sparsity, but the order is the same.</p>\n<p>Speed is the other half of the case for 2:4, and the lab timed a 2:4 sparse weight matrix of the Llama 3 70B MLP shape, 8,192 by 28,672, against the same matrix multiplied densely in BF16:</p>\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n<div class=\"table-wrap\"><table><thead><tr><th>Rows of activations</th><th>16</th><th>64</th><th>256</th><th>1,024</th><th>4,096</th><th>8,192</th></tr></thead><tbody><tr><td>H100, cuSPARSELt</td><td>0.31×</td><td>0.35×</td><td>0.54×</td><td>0.61×</td><td>0.74×</td><td>0.76×</td></tr><tr><td>L4, cuSPARSELt</td><td>1.59×</td><td>1.69×</td><td>0.64×</td><td>0.48×</td><td>0.44×</td><td>0.43×</td></tr><tr><td>L4, CUTLASS</td><td>1.63×</td><td>1.72×</td><td>1.18×</td><td>0.87×</td><td>0.81×</td><td>0.79×</td></tr></tbody></table></div>\n<p>On the H100, PyTorch's 2:4 path was slower than the dense matmul at every size, and its CUTLASS path supports only compute capability 8.x. On the L4 it helped only at small batches, where the matmul is memory-bound and the sparse matrix is 252 MiB against 448: the gain came from reading fewer bytes, not from the sparse tensor cores' arithmetic, and at large batches it lost. The hardware's 2 times is real for well-tuned kernels at suitable shapes, as NVIDIA's measurements show, but a general library may not deliver it.</p>","headings":[{"id":"measured-half-the-weights-of-a-small-model","title":"Measured: half the weights of a small model","depth":3}]},{"id":"distillation","title":"Distillation","html":"<h3 id=\"distillation\">Distillation</h3>\n<p>Knowledge distillation, named by Hinton, Vinyals and Dean in 2015, trains a smaller student model to match a larger teacher's output distributions rather than only the correct next token. The teacher's probabilities carry more information per example than a single label: they say which wrong answers are nearly right. DistilBERT, an early example, was 40% smaller than BERT, ran 60% faster, and kept 97% of its language understanding.</p>\n<p>For LLMs, distillation is now combined with structured pruning. NVIDIA's Minitron work pruned the 15B Nemotron-4 model's attention heads, MLP neurons and embedding channels to make 8B and 4B models, then retrained them by distillation from the original, using up to 40 times fewer training tokens than training those sizes from scratch. Meta made Llama 3.2 1B and 3B by structured pruning of Llama 3.1 8B, then used the logits of Llama 3.1 8B and 70B as targets during their pre-training. The result of distillation is an ordinary dense model, so it runs on every engine and every GPU, and it can be quantized afterwards like any other.</p>\n<p>The cost is training. Pruning and distillation of this kind takes tens of billions to trillions of tokens on a GPU cluster, so it is done by model producers, not by a serving team. A serving team's version of the choice is simpler: use the smaller distilled model if one exists and is good enough, and quantize it.</p>","headings":[{"id":"distillation","title":"Distillation","depth":3}]},{"id":"when-each-is-worth-it","title":"When each is worth it","html":"<h3 id=\"when-each-is-worth-it\">When each is worth it</h3>\n<p>Quantization is the default because it is cheap and reliable: post-training methods run in minutes to an hour on one GPU, every serving engine supports the common formats, and at 8 bits the quality loss is usually negligible. Pruning to 2:4 needs its own kernels, gives at most a 2 times speedup on the matrix multiplications and, as the lab shows, often less in practice, and it costs more quality than INT4 quantization at a similar memory saving unless the model is retrained. Unstructured sparsity saves little or nothing on GPUs with common libraries at the levels that keep quality; research kernels gain only at much higher sparsity. Distillation gives the largest, most portable savings, a genuinely smaller model, but at the price of a training run. In practice, a serving team chooses among the distilled sizes model producers publish, then quantizes the one it picks.</p>","headings":[{"id":"when-each-is-worth-it","title":"When each is worth it","depth":3}]},{"id":"lab-prune-a-model-then-time-sparse-matmuls","title":"Lab: prune a model, then time sparse matmuls","html":"<h3 id=\"lab-prune-a-model-then-time-sparse-matmuls\">Lab: prune a model, then time sparse matmuls</h3>\n<p><strong>Tier: [Mac] then [1 GPU].</strong> Part A runs on a CPU in about seven minutes and needs the lab files from chapters 4.2 and 4.3 in the same folder. Part B needs an NVIDIA GPU of compute capability 8.0 or later.</p>\n<p>Download <a href=\"/labs/infra/pruning.py\">pruning.py</a> and <a href=\"/labs/infra/gpu/sparse_matmul.py\">gpu/sparse_matmul.py</a>. Run the first with <code>python pruning.py</code>, and the second from the <code>gpu/</code> folder:</p>\n<pre class=\"shiki shiki-themes github-light github-dark\" style=\"background-color:#fff;--shiki-dark-bg:#24292e;color:#24292e;--shiki-dark:#e1e4e8\" tabindex=\"0\"><code><span class=\"line\"><span style=\"color:#6F42C1;--shiki-dark:#B392F0\">modal</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> run</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> modal_run.py</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> --script</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> sparse_matmul.py</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> --gpu</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> 'H100!'</span><span style=\"color:#6A737D;--shiki-dark:#6A737D\">  # about 10 s of GPU time, under $0.01</span></span>\n<span class=\"line\"><span style=\"color:#6F42C1;--shiki-dark:#B392F0\">python</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> gcp_run.py</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> --script</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> sparse_matmul.py</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> --gpu</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> L4</span><span style=\"color:#6A737D;--shiki-dark:#6A737D\">            # a Spot VM; about $0.12 with the image build</span></span></code></pre>\n<p><strong>Part A, pruning.</strong> Prune every linear layer of SmolLM2-135M to 50% sparsity by magnitude and by Wanda's score, unstructured and 2:4, and compare the perplexities with each other and with chapter 4.3's INT4 results.</p>\n<p><strong>Part B, sparse matmuls.</strong> Convert a pruned 2:4 weight matrix with <code>torch.sparse.to_sparse_semi_structured</code> and time it against the dense matmul at decode and prefill shapes. Explain why the speedup is below 2 times, or why there is none.</p>\n<details>\n<summary>Reference solution</summary>\n<p><code>pruning.py</code> on an Apple M2 MacBook with 8 GB of memory:</p>\n<pre class=\"shiki shiki-themes github-light github-dark\" style=\"background-color:#fff;--shiki-dark-bg:#24292e;color:#24292e;--shiki-dark:#e1e4e8\" tabindex=\"0\"><code><span class=\"line\"><span>PyTorch 2.15.0, Transformers 5.22.2</span></span>\n<span class=\"line\"><span></span></span>\n<span class=\"line\"><span>WikiText-2 test perplexity, every linear layer pruned to 50% sparsity (106.2M weights)</span></span>\n<span class=\"line\"><span>   method                          perplexity   change</span></span>\n<span class=\"line\"><span>   dense                                17.10</span></span>\n<span class=\"line\"><span>   magnitude, unstructured             466.04   +2625%</span></span>\n<span class=\"line\"><span>   Wanda, unstructured                  36.28    +112%</span></span>\n<span class=\"line\"><span>   magnitude, 2:4                     1477.17   +8537%</span></span>\n<span class=\"line\"><span>   Wanda, 2:4                          104.63    +512%</span></span>\n<span class=\"line\"><span></span></span>\n<span class=\"line\"><span>   For comparison, chapter 4.3: INT4 group 128 with GPTQ (act-order) 20.09 (+17%), with RTN 23.01 (+35%).</span></span>\n<span class=\"line\"><span>   2:4 sparse BF16 stores 9 bits per original weight (two 16-bit values and two 2-bit indices per four);</span></span>\n<span class=\"line\"><span>   INT4 with group-128 scales stores about 4.2.</span></span></code></pre>\n<p><em>Part A.</em> Wanda's activation-aware score is what keeps a pruned model working: magnitude pruning removes small weights on channels with large inputs, and those weights carry much of the output. The 2:4 pattern costs more than unstructured pruning at the same sparsity, because it must remove two weights from every group of four, including groups whose four weights all matter. Even the best pruned version is far worse than INT4 quantization while storing more than twice the bits.</p>\n<p><code>sparse_matmul.py</code> on Modal's H100 and on a Google Cloud Spot L4 (g2-standard-8):</p>\n<pre class=\"shiki shiki-themes github-light github-dark\" style=\"background-color:#fff;--shiki-dark-bg:#24292e;color:#24292e;--shiki-dark:#e1e4e8\" tabindex=\"0\"><code><span class=\"line\"><span>NVIDIA H100 80GB HBM3, PyTorch 2.15.0+cu132, CUDA 13.2</span></span>\n<span class=\"line\"><span></span></span>\n<span class=\"line\"><span>cusparselt: dense BF16 weights 448 MiB; 2:4 values + indices 252 MiB</span></span>\n<span class=\"line\"><span>        M      dense  2:4 sparse  speedup  max diff</span></span>\n<span class=\"line\"><span>       16    155.2us     497.2us    0.31x   2.5e-03</span></span>\n<span class=\"line\"><span>       64    163.2us     467.0us    0.35x   4.7e-03</span></span>\n<span class=\"line\"><span>      256    184.9us     340.8us    0.54x   4.3e-03</span></span>\n<span class=\"line\"><span>     1024    582.9us     955.3us    0.61x   4.3e-03</span></span>\n<span class=\"line\"><span>     4096   2712.6us    3672.8us    0.74x   4.0e-03</span></span>\n<span class=\"line\"><span>     8192   5618.7us    7355.0us    0.76x   4.0e-03</span></span>\n<span class=\"line\"><span></span></span>\n<span class=\"line\"><span>cutlass: (not supported: \"Supported only on GPUs with compute capability 8.x\")</span></span>\n<span class=\"line\"><span></span></span>\n<span class=\"line\"><span>NVIDIA L4, PyTorch 2.15.0+cu132, CUDA 13.2</span></span>\n<span class=\"line\"><span></span></span>\n<span class=\"line\"><span>cusparselt: dense BF16 weights 448 MiB; 2:4 values + indices 252 MiB</span></span>\n<span class=\"line\"><span>        M      dense  2:4 sparse  speedup  max diff</span></span>\n<span class=\"line\"><span>       16   1894.6us    1192.9us    1.59x   4.2e-03</span></span>\n<span class=\"line\"><span>       64   2073.1us    1225.4us    1.69x   2.3e-03</span></span>\n<span class=\"line\"><span>      256   3028.6us    4763.2us    0.64x   4.3e-03</span></span>\n<span class=\"line\"><span>     1024   9104.2us   18818.0us    0.48x   4.3e-03</span></span>\n<span class=\"line\"><span>     4096  32938.5us   75134.7us    0.44x   4.1e-03</span></span>\n<span class=\"line\"><span>     8192  65357.0us  150273.2us    0.43x   4.1e-03</span></span>\n<span class=\"line\"><span></span></span>\n<span class=\"line\"><span>cutlass: dense BF16 weights 448 MiB; 2:4 values + indices 252 MiB</span></span>\n<span class=\"line\"><span>        M      dense  2:4 sparse  speedup  max diff</span></span>\n<span class=\"line\"><span>       16   1897.6us    1164.7us    1.63x   2.5e-03</span></span>\n<span class=\"line\"><span>       64   2108.0us    1222.6us    1.72x   4.8e-03</span></span>\n<span class=\"line\"><span>      256   3236.1us    2749.8us    1.18x   4.5e-03</span></span>\n<span class=\"line\"><span>     1024   9255.3us   10609.3us    0.87x   3.9e-03</span></span>\n<span class=\"line\"><span>     4096  34343.0us   42581.6us    0.81x   4.0e-03</span></span>\n<span class=\"line\"><span>     8192  68144.6us   86761.9us    0.79x   3.8e-03</span></span></code></pre>\n<p><em>Part B.</em> The sparse matmul is correct (its result matches the dense product of the same pruned matrix to BF16 precision), but the speedup depends on the regime. At 16 to 64 rows on the L4, the matmul is memory-bound and the sparse matrix has 56% of the dense one's bytes, which allows at most 1.8 times; the lab measured 1.6 to 1.7 times, so the gain is bandwidth. At large batches the matmul is compute-bound, where the sparse tensor cores could double the arithmetic rate, but the sparse kernels PyTorch reaches through cuSPARSELt and CUTLASS ran below the dense cuBLAS kernels on both GPUs. On the H100, whose dense kernels are the most heavily tuned, the sparse path lost everywhere. A 2:4 speedup has to be measured on the target GPU, library version and shapes; it cannot be assumed from the datasheet.</p>\n</details>","headings":[{"id":"lab-prune-a-model-then-time-sparse-matmuls","title":"Lab: prune a model, then time sparse matmuls","depth":3}]},{"id":"interview-angle","title":"Interview angle","html":"<h3 id=\"interview-angle\">Interview angle</h3>\n<ul>\n<li><strong>\"What is 2:4 sparsity?\"</strong> A pattern in which every group of four consecutive weights has at least two zeros, which NVIDIA's sparse tensor cores (Ampere and later) can skip, storing half the values plus 2-bit indices. It promises up to twice the matmul throughput, but only for that exact pattern at 50% sparsity, and it costs more accuracy than unstructured pruning.</li>\n<li><strong>\"How does Wanda decide what to prune?\"</strong> It scores each weight by its magnitude times the L2 norm of its input channel's activations over calibration data and removes the lowest-scoring weights in each output row, with no weight update. Including the activations protects weights on channels with large inputs, which magnitude pruning ignores.</li>\n<li><strong>\"Why is quantization used far more than pruning for LLM serving?\"</strong> It is cheaper to apply, needs no retraining, has broad kernel support in every engine, and halves or quarters memory and bandwidth at small quality cost. Unstructured sparsity gives little or no GPU speedup with common libraries at the levels that keep quality, 2:4 sparsity needs special kernels and gives less than 2 times in practice, and recovering quality after pruning usually needs training.</li>\n<li><strong>\"What is distillation, and who does it?\"</strong> Training a small student model to match a large teacher's output distributions. It produces a dense smaller model that runs anywhere, and model producers combine it with structured pruning, as in NVIDIA's Minitron models and Meta's Llama 3.2 1B and 3B; a serving team usually consumes distilled models rather than making them.</li>\n</ul>","headings":[{"id":"interview-angle","title":"Interview angle","depth":3}]},{"id":"check-yourself","title":"Check yourself","html":"<h3 id=\"check-yourself\">Check yourself</h3>\n<details>\n<summary>Why does 50% unstructured sparsity not save memory in a typical sparse format?</summary>\n<p>Each kept value needs an index saying where it is. With 16-bit values and 16- or 32-bit column indices, a matrix with half its values kept takes as much memory as the dense one or more. Only the 2:4 pattern, whose position within a group of four fits in 2 bits, stores compactly: about 9 bits per original weight in BF16.</p>\n</details>\n<details>\n<summary>A 2:4 sparse matmul does half the multiplications. Why can it be slower than the dense one?</summary>\n<p>At small batch sizes the matmul is memory-bound, and the sparse matrix is still about 56% of the dense one's bytes, so at best it saves a little bandwidth, while its kernel must also read and decode the index metadata. At large batch sizes it is compute-bound, where the sparse tensor cores can help, but only if the library has a well-tuned sparse kernel for that GPU and shape; otherwise the dense library's highly tuned kernels win.</p>\n</details>\n<details>\n<summary>What do pruning a model and quantizing it to INT4 each do to its memory, at 50% sparsity in BF16 and with group-128 INT4?</summary>\n<p>2:4 sparsity in BF16 stores about 9 bits per original weight, a 44% saving over BF16's 16. INT4 with group-128 FP16 scales stores about 4.1 to 4.2 bits per weight, a 74% saving, and in the labs it lost less quality.</p>\n</details>\n<h3 id=\"further-reading\">Further reading</h3>\n<ul>\n<li><a href=\"https://arxiv.org/abs/2104.08378\" target=\"_blank\" rel=\"noopener noreferrer\">Accelerating Sparse Deep Neural Networks</a>, by Mishra and others at NVIDIA, describes 2:4 sparsity and the sparse tensor cores.</li>\n<li><a href=\"https://arxiv.org/abs/2301.00774\" target=\"_blank\" rel=\"noopener noreferrer\">SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot</a> and <a href=\"https://arxiv.org/abs/2306.11695\" target=\"_blank\" rel=\"noopener noreferrer\">A Simple and Effective Pruning Approach for Large Language Models</a> (Wanda).</li>\n<li><a href=\"https://arxiv.org/abs/1503.02531\" target=\"_blank\" rel=\"noopener noreferrer\">Distilling the Knowledge in a Neural Network</a>, by Hinton, Vinyals and Dean, and <a href=\"https://arxiv.org/abs/1910.01108\" target=\"_blank\" rel=\"noopener noreferrer\">DistilBERT</a>.</li>\n<li><a href=\"https://arxiv.org/abs/2407.14679\" target=\"_blank\" rel=\"noopener noreferrer\">Compact Language Models via Pruning and Knowledge Distillation</a>, NVIDIA's Minitron paper.</li>\n<li>Meta's <a href=\"https://ai.meta.com/blog/llama-3-2-connect-2024-vision-edge-mobile-devices/\" target=\"_blank\" rel=\"noopener noreferrer\">Llama 3.2 announcement</a> describes how the 1B and 3B models were pruned and distilled.</li>\n<li>PyTorch's <a href=\"https://docs.pytorch.org/tutorials/advanced/semi_structured_sparse.html\" target=\"_blank\" rel=\"noopener noreferrer\">semi-structured sparsity tutorial</a> shows <code>to_sparse_semi_structured</code> on a BERT model.</li>\n</ul>","headings":[{"id":"check-yourself","title":"Check yourself","depth":3},{"id":"further-reading","title":"Further reading","depth":3}]}]}