{"id":"models-and-machines","title":"Models and machines","sections":[{"id":"overview","title":"Overview","html":"<p>Numbers only make sense against a fixed model and a fixed machine. This handbook uses three models and three machines throughout, so a figure in Part 5 can be checked against one in Part 1. This chapter introduces them, shows how to read a model's shape from its <code>config.json</code>, and derives the three numbers every later chapter needs: parameter count, weight bytes, and KV cache bytes per token.</p>\n<p>Before this chapter, read <a href=\"/learn/llm-inference#the-kv-cache\">the KV cache</a> and <a href=\"/learn/gpus-for-ai#gpu-memory\">GPU memory</a>.</p>","headings":[]},{"id":"the-three-models","title":"The three models","html":"<h3 id=\"the-three-models\">The three models</h3>\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n<div class=\"table-wrap\"><table><thead><tr><th>Model</th><th>Why it is here</th><th>Where its labs run</th></tr></thead><tbody><tr><td><strong>Llama 3.2 1B</strong></td><td>Small enough for a laptop, same architecture as its larger siblings</td><td>[Mac]</td></tr><tr><td><strong>Llama 3.1 8B</strong></td><td>The main running example: the size many teams serve, and it fits one GPU</td><td>[1 GPU]</td></tr><tr><td><strong>Qwen3-30B-A3B</strong></td><td>A mixture-of-experts (MoE) model: 30B parameters stored, about 3B used per token</td><td>[1 GPU], H100</td></tr></tbody></table></div>\n<p>All three are decoder-only transformers with grouped-query attention, SwiGLU MLPs and rotary position embeddings, so one set of formulas covers them, with one change for the MoE layers. The system design chapters use Llama 3.1 70B. This handbook moves to 8B because it fits one rented GPU, which makes the labs cheap.</p>","headings":[{"id":"the-three-models","title":"The three models","depth":3}]},{"id":"reading-configjson","title":"Reading config.json","html":"<h3 id=\"reading-configjson\">Reading config.json</h3>\n<p>Every model on Hugging Face ships a <code>config.json</code> that states its shape. These are the fields that matter for serving, with Llama 3.1 8B's values:</p>\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n<div class=\"table-wrap\"><table><thead><tr><th>Field</th><th>8B value</th><th>Meaning</th></tr></thead><tbody><tr><td><code>hidden_size</code></td><td>4,096</td><td>Width d of the vector that carries each token between layers</td></tr><tr><td><code>num_hidden_layers</code></td><td>32</td><td>Number of decoder layers, L</td></tr><tr><td><code>num_attention_heads</code></td><td>32</td><td>Query heads, H</td></tr><tr><td><code>num_key_value_heads</code></td><td>8</td><td>Key/value heads, KV. Fewer than H means grouped-query attention</td></tr><tr><td><code>head_dim</code></td><td>128</td><td>Width of each head. Often d ÷ H, but not always</td></tr><tr><td><code>intermediate_size</code></td><td>14,336</td><td>Width of the MLP's hidden layer, I</td></tr><tr><td><code>vocab_size</code></td><td>128,256</td><td>Number of tokens the model knows, V</td></tr><tr><td><code>tie_word_embeddings</code></td><td>false</td><td>Whether the output layer reuses the input embedding matrix</td></tr><tr><td><code>max_position_embeddings</code></td><td>131,072</td><td>Longest context the model was built for</td></tr><tr><td><code>rope_scaling</code></td><td>llama3, factor 8</td><td>How rotary positions were stretched for long context</td></tr><tr><td><code>torch_dtype</code></td><td>bfloat16</td><td>The precision the weights ship in</td></tr></tbody></table></div>\n<p>The MoE model adds <code>num_experts</code> (128), <code>num_experts_per_tok</code> (8) and <code>moe_intermediate_size</code> (768): each layer holds 128 small MLPs and sends each token through 8 of them.</p>\n<p>Configs vary by family. Some omit <code>head_dim</code>, some name fields differently, and models with other attention schemes (DeepSeek's latent attention, sliding windows) add fields the formulas below do not cover. Read the config before trusting any calculator, including the one in this chapter's lab.</p>","headings":[{"id":"reading-configjson","title":"Reading config.json","depth":3}]},{"id":"counting-parameters","title":"Counting parameters","html":"<h3 id=\"counting-parameters\">Counting parameters</h3>\n<p>A Llama-style layer has four attention matrices and three MLP matrices. With the 8B values, they break down as follows:</p>\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n<div class=\"table-wrap\"><table><thead><tr><th>Piece</th><th>Shape</th><th>Parameters</th></tr></thead><tbody><tr><td>Q projection</td><td>d × (H × head_dim) = 4,096 × 4,096</td><td>16.8M</td></tr><tr><td>K projection</td><td>d × (KV × head_dim) = 4,096 × 1,024</td><td>4.2M</td></tr><tr><td>V projection</td><td>4,096 × 1,024</td><td>4.2M</td></tr><tr><td>Output projection</td><td>4,096 × 4,096</td><td>16.8M</td></tr><tr><td>MLP gate, up, down</td><td>3 × 4,096 × 14,336</td><td>176.2M</td></tr><tr><td>Two RMSNorm weights</td><td>2 × 4,096</td><td>8K</td></tr><tr><td><strong>One layer</strong></td><td></td><td><strong>218.1M</strong></td></tr></tbody></table></div>\n<p>Thirty-two layers hold 6.98B parameters. The input embedding (V × d = 525.3M) and the separate output layer, the LM head (another 525.3M), bring the total to 8.03B.</p>\n<p>Two counts matter, and they differ:</p>\n<ul>\n<li><strong>Stored parameters</strong>, 8.03B for this model, decide memory.</li>\n<li><strong>Parameters in matmuls per token</strong>, 7.50B, decide compute. The input embedding is a table lookup, which copies one row per token instead of multiplying, so it is left out.</li>\n</ul>\n<p>Each matmul parameter costs one multiply and one add per token, so a token costs about 2 × 7.50B = 15.0 GFLOP, before attention scores (the next chapter adds those).</p>\n<p><strong>Tied embeddings.</strong> Llama 3.2 1B sets <code>tie_word_embeddings</code> to true, so the LM head reuses the embedding matrix. The 262.7M-parameter table is stored once but used in a matmul at the output, so stored and per-token counts are both 1.24B.</p>\n<p><strong>Mixture of experts.</strong> In Qwen3-30B-A3B each layer's MLP is 128 experts of 3 × 2,048 × 768 = 4.7M parameters, plus a 2,048 × 128 router that scores the experts. The stored total is 48 layers × (18.9M attention + 604M experts + 0.26M router) plus embeddings, which comes to 30.53B. Per token, only 8 experts run, so 3.04B parameters take part in matmuls. Qwen's model card says 3.3B activated; the gap is the 0.31B input embedding, which this chapter counts as a lookup. Memory is sized by the 30.5B, compute by the 3B. Part 1's MoE chapter returns to what that does to serving.</p>","headings":[{"id":"counting-parameters","title":"Counting parameters","depth":3}]},{"id":"weights-in-bytes","title":"Weights in bytes","html":"<h3 id=\"weights-in-bytes\">Weights in bytes</h3>\n<p>Weight memory is stored parameters times bytes per parameter:</p>\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n<div class=\"table-wrap\"><table><thead><tr><th>Model</th><th>BF16 (2 bytes)</th><th>FP8 (1 byte)</th><th>INT4 (0.5 bytes)</th></tr></thead><tbody><tr><td>Llama 3.2 1B</td><td>2.5 GB</td><td>1.2 GB</td><td>0.6 GB</td></tr><tr><td>Llama 3.1 8B</td><td>16.1 GB</td><td>8.0 GB</td><td>4.0 GB</td></tr><tr><td>Qwen3-30B-A3B</td><td>61.1 GB</td><td>30.5 GB</td><td>15.3 GB</td></tr></tbody></table></div>\n<p>The INT4 column is a floor. Real 4-bit formats store a scale for each small group of weights. One 16-bit scale per 128 weights adds 16 ÷ 128 = 0.125 bits per weight, and many tools leave the embeddings and LM head in 16 bits, so a \"4-bit\" checkpoint is usually larger than half a byte per parameter. Part 4 covers the formats.</p>","headings":[{"id":"weights-in-bytes","title":"Weights in bytes","depth":3}]},{"id":"kv-cache-per-token","title":"KV cache per token","html":"<h3 id=\"kv-cache-per-token\">KV cache per token</h3>\n<p>Each layer stores one key and one value vector per KV head for every token in the context:</p>\n<p>KV bytes per token = 2 (K and V) × layers × KV heads × head_dim × bytes per value</p>\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n<div class=\"table-wrap\"><table><thead><tr><th>Model</th><th>Calculation in BF16</th><th>Bytes per token</th><th>One 8K-token sequence</th></tr></thead><tbody><tr><td>Llama 3.2 1B</td><td>2 × 16 × 8 × 64 × 2</td><td>32,768</td><td>0.27 GB</td></tr><tr><td>Llama 3.1 8B</td><td>2 × 32 × 8 × 128 × 2</td><td>131,072</td><td>1.07 GB</td></tr><tr><td>Qwen3-30B-A3B</td><td>2 × 48 × 4 × 128 × 2</td><td>98,304</td><td>0.81 GB</td></tr></tbody></table></div>\n<p>Grouped-query attention is why these are small. If the 8B model had 32 KV heads, one per query head, its cache would be 524,288 bytes per token, four times larger. The MoE model is 3.8 times bigger than the 8B on disk but needs less cache per token, because it has only 4 KV heads. The cache depends on the attention layout, not the parameter count.</p>\n<p>Try it with the larger Llama models: pick a model, a GPU and a context length, and see how many requests fit.</p>\n<div class=\"widget-slot\" data-widget=\"kv-cache-calculator\"></div>","headings":[{"id":"kv-cache-per-token","title":"KV cache per token","depth":3}]},{"id":"the-three-machines","title":"The three machines","html":"<h3 id=\"the-three-machines\">The three machines</h3>\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n<div class=\"table-wrap\"><table><thead><tr><th>Machine</th><th>Memory</th><th>Bandwidth</th><th>Dense BF16 or FP16</th><th>Ridge point</th><th>Role here</th></tr></thead><tbody><tr><td>NVIDIA H100 SXM</td><td>80 GB HBM3</td><td>3.35 TB/s</td><td>989 TFLOPS</td><td>295 FLOPs/byte</td><td>The reference numbers, and the FP8 labs</td></tr><tr><td>NVIDIA L4</td><td>24 GB GDDR6</td><td>300 GB/s</td><td>121 TFLOPS</td><td>403 FLOPs/byte</td><td>The cheap rental for most [1 GPU] labs</td></tr><tr><td>Apple M2, 8-core GPU, 8 GB</td><td>8 GB unified, 5.7 GB usable by the GPU</td><td>100 GB/s</td><td>2.5 TFLOPS FP16, measured</td><td>about 30 FLOPs/byte</td><td>The laptop for [Mac] labs</td></tr></tbody></table></div>\n<p>The figures come from these sources:</p>\n<ul>\n<li>The H100 figures and their sparsity footnote are explained in <a href=\"/learn/gpus-for-ai#gpu-memory\">GPUs and accelerators</a>.</li>\n<li>NVIDIA's L4 datasheet prints 242 BF16 teraFLOPS \"with sparsity\" and says dense figures are half of that, so the dense figure is 121 TFLOPS.</li>\n<li>Apple gives the M2 100 GB/s of unified memory bandwidth. Apple does not publish an FP16 matmul figure for the 8-core GPU, so the 2.5 TFLOPS is measured with the probe in <a href=\"/learn/inference-arithmetic#measuring-your-own-roofline\">the arithmetic of inference</a>, which also measured 84 GB/s for a large copy. The 5.7 GB is what <code>torch.mps.recommended_max_memory()</code> reported on that machine; macOS keeps the rest for itself.</li>\n</ul>\n<p>The ridge point is peak FLOPs divided by bandwidth: the arithmetic intensity at which an operation stops being limited by memory. <a href=\"/learn/gpus-for-ai#compute-versus-memory-bandwidth\">Compute versus memory bandwidth</a> explains it, and Part 1 uses it constantly. Note the L4: it has less bandwidth per FLOP than an H100, so decode, which is bandwidth-bound, is relatively slower on it.</p>","headings":[{"id":"the-three-machines","title":"The three machines","depth":3}]},{"id":"what-fits-where","title":"What fits where","html":"<h3 id=\"what-fits-where\">What fits where</h3>\n<p>The table takes usable memory as 90% of GPU memory, keeping 10% for activations and the runtime as the system design chapters do. On the M2 it uses the 5.7 GB the GPU can address. The KV cache gets whatever the weights leave.</p>\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n<div class=\"table-wrap\"><table><thead><tr><th>Model and format</th><th>M2 (5.7 GB)</th><th>L4 (21.6 GB)</th><th>H100 (72 GB)</th></tr></thead><tbody><tr><td>1B, BF16 (2.5 GB)</td><td>fits, 3.2 GB for cache, about 98K tokens</td><td>fits, 584K tokens</td><td>fits</td></tr><tr><td>8B, BF16 (16.1 GB)</td><td>no</td><td>fits, 5.5 GB for cache, about 42K tokens</td><td>fits, 55.9 GB, about 427K tokens</td></tr><tr><td>8B, FP8 (8.0 GB)</td><td>no</td><td>fits, about 104K tokens</td><td>fits, about 488K tokens</td></tr><tr><td>8B, INT4 (4.0 GB and up)</td><td>barely; little room for cache</td><td>fits</td><td>fits</td></tr><tr><td>30B-A3B, BF16 (61.1 GB)</td><td>no</td><td>no</td><td>fits, 10.9 GB, about 111K tokens</td></tr><tr><td>30B-A3B, FP8 (30.5 GB)</td><td>no</td><td>no</td><td>fits, about 422K tokens</td></tr><tr><td>30B-A3B, INT4 (15.3 GB and up)</td><td>no</td><td>fits, about 64K tokens</td><td>fits</td></tr></tbody></table></div>\n<p>Three lessons come out of the table:</p>\n<ul>\n<li><strong>Cache room, not weights, decides how many users fit.</strong> The 8B model fits an L4 in BF16, but 42K tokens of cache is about 20 conversations of 2,000 tokens.</li>\n<li><strong>Quantizing weights buys cache.</strong> FP8 weights on the L4 leave room for about 2.5 times as many cached tokens as BF16.</li>\n<li><strong>MoE models are memory-hungry.</strong> Qwen3-30B-A3B computes like a 3B model but needs a big GPU, or heavy quantization, just to load.</li>\n</ul>","headings":[{"id":"what-fits-where","title":"What fits where","depth":3}]},{"id":"lab-a-model-calculator","title":"Lab: a model calculator","html":"<h3 id=\"lab-a-model-calculator\">Lab: a model calculator</h3>\n<p><strong>Tier: [any].</strong> This lab runs on any machine. You write the tool that later labs use to size models.</p>\n<p>Write <code>labs/model_stats.py</code>. Given one or more paths to <code>config.json</code> files, it should print the following for each model:</p>\n<ol>\n<li>layers, hidden size, query heads, KV heads and head dimension;</li>\n<li>stored parameters and parameters in matmuls per token;</li>\n<li>weight bytes in BF16, FP8 and INT4;</li>\n<li>FLOPs per token, ignoring attention scores;</li>\n<li>KV cache bytes per token, and for one 8K-token sequence.</li>\n</ol>\n<p>Download the configs with <code>curl -L https://huggingface.co/&#x3C;repo>/raw/main/config.json</code>. Meta's own Llama repositories need you to accept a licence first; mirrors such as <code>unsloth/Llama-3.2-1B-Instruct</code> and <code>unsloth/Meta-Llama-3.1-8B-Instruct</code> publish the same configs. Qwen's is at <code>Qwen/Qwen3-30B-A3B</code>.</p>\n<p>The lab is done when your parameter counts are within 1% of the model cards (1.23B, 8.03B, 30.5B total) and the KV numbers match the table above.</p>\n<p>As a stretch goal, run it on <code>deepseek-ai/DeepSeek-V3</code>'s config. Its KV numbers will come out wrong, because DeepSeek's multi-head latent attention caches a compressed vector instead of per-head keys and values. Read the config fields <code>kv_lora_rank</code> and <code>qk_rope_head_dim</code> and work out what it really caches per token per layer; <a href=\"/learn/transformer-as-workload#attention-variants\">the next chapter</a> describes the scheme.</p>\n<p>The reference solution can be downloaded as <a href=\"/labs/infra/model_stats.py\">model_stats.py</a>, together with the three configs it was tested on: <a href=\"/labs/infra/configs/Llama-3.2-1B-Instruct.json\">Llama 3.2 1B</a>, <a href=\"/labs/infra/configs/Meta-Llama-3.1-8B-Instruct.json\">Llama 3.1 8B</a> and <a href=\"/labs/infra/configs/Qwen3-30B-A3B.json\">Qwen3-30B-A3B</a>. Run it as <code>python model_stats.py *.json</code> in the folder that holds them.</p>\n<details>\n<summary>Reference solution</summary>\n<pre class=\"shiki shiki-themes github-light github-dark\" style=\"background-color:#fff;--shiki-dark-bg:#24292e;color:#24292e;--shiki-dark:#e1e4e8\" tabindex=\"0\"><code><span class=\"line\"><span style=\"color:#6A737D;--shiki-dark:#6A737D\"># model_stats.py: parameter count, weight bytes and KV cache bytes from a Hugging Face config.json.</span></span>\n<span class=\"line\"><span style=\"color:#6A737D;--shiki-dark:#6A737D\"># Handles Llama-style dense models and Qwen3-style mixture-of-experts models.</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">import</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> json</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">import</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> sys</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">BYTES</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> =</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> {</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"BF16\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">2</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"FP8\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">1</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"INT4\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">0.5</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">}</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">def</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\"> stats</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(cfg: </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">dict</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">) -> </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">dict</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">:</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">    d </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> cfg[</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"hidden_size\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">]</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">    layers </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> cfg[</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"num_hidden_layers\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">]</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">    heads </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> cfg[</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"num_attention_heads\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">]</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">    kv_heads </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> cfg.get(</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"num_key_value_heads\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, heads)</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">    head_dim </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> cfg.get(</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"head_dim\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">) </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">or</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> d </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">//</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> heads</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">    vocab </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> cfg[</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"vocab_size\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">]</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#6A737D;--shiki-dark:#6A737D\">    # Attention: Q and O are heads x head_dim wide, K and V are kv_heads x head_dim wide.</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">    attn </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> d </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">*</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> heads </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">*</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> head_dim </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">+</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 2</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> *</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> d </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">*</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> kv_heads </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">*</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> head_dim </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">+</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> heads </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">*</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> head_dim </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">*</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> d</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    if</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> cfg[</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"model_type\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">].startswith(</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"qwen3\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">):</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">        attn </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">+=</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 2</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> *</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> head_dim  </span><span style=\"color:#6A737D;--shiki-dark:#6A737D\"># Qwen3 normalizes each query and key head (q_norm, k_norm)</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#6A737D;--shiki-dark:#6A737D\">    # MLP: gate, up and down projections. An MoE layer has one set per expert plus a router.</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    if</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> \"num_experts\"</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> in</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> cfg:</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">        experts, top_k </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> cfg[</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"num_experts\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">], cfg[</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"num_experts_per_tok\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">]</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">        expert </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 3</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> *</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> d </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">*</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> cfg[</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"moe_intermediate_size\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">]</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">        router </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> d </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">*</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> experts</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">        mlp_total, mlp_active </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> experts </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">*</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> expert </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">+</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> router, top_k </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">*</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> expert </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">+</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> router</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    else</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">:</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">        mlp_total </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> mlp_active </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 3</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> *</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> d </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">*</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> cfg[</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"intermediate_size\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">]</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">    norms </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 2</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> *</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> d  </span><span style=\"color:#6A737D;--shiki-dark:#6A737D\"># two RMSNorms per layer</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">    embed </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> vocab </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">*</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> d</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">    head </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 0</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> if</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> cfg.get(</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"tie_word_embeddings\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">) </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">else</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> vocab </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">*</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> d</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">    total </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> embed </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">+</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> head </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">+</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> layers </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">*</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> (attn </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">+</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> mlp_total </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">+</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> norms) </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">+</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> d</span></span>\n<span class=\"line\"><span style=\"color:#6A737D;--shiki-dark:#6A737D\">    # Matmul weights one token multiplies through (the embedding lookup is a copy, not a matmul).</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">    active_matmul </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> layers </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">*</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> (attn </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">+</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> mlp_active) </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">+</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> vocab </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">*</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> d</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    return</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> {</span></span>\n<span class=\"line\"><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">        \"layers\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: layers, </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"d\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: d, </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"heads\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: heads, </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"kv_heads\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: kv_heads, </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"head_dim\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: head_dim,</span></span>\n<span class=\"line\"><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">        \"total\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: total, </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"active_matmul\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: active_matmul,</span></span>\n<span class=\"line\"><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">        \"kv_bytes_per_token\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">2</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> *</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> layers </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">*</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> kv_heads </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">*</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> head_dim </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">*</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 2</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">,  </span><span style=\"color:#6A737D;--shiki-dark:#6A737D\"># K and V, BF16</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">    }</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">def</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\"> report</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(path: </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">str</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">) -> </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">None</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">:</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">    cfg </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> json.load(</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">open</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(path))</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">    s </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> stats(cfg)</span></span>\n<span class=\"line\"><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">    print</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">f</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">{</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">path.split(</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">'/'</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">)[</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">-</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">1</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">]</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">}</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">: </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">{</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">s[</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">'layers'</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">]</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">}</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> layers, d=</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">{</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">s[</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">'d'</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">]</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">}</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">, \"</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">          f</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">{</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">s[</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">'heads'</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">]</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">}</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> heads / </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">{</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">s[</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">'kv_heads'</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">]</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">}</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> KV heads of </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">{</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">s[</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">'head_dim'</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">]</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">}</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">)</span></span>\n<span class=\"line\"><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">    print</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">f</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"  parameters: </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">{</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">s[</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">'total'</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">] </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">/</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 1e9</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">:.2f</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">}</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">B stored, \"</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">          f</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">{</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">s[</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">'active_matmul'</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">] </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">/</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 1e9</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">:.2f</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">}</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">B in matmuls per token\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">)</span></span>\n<span class=\"line\"><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">    print</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"  weights: \"</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> +</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> \", \"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">.join(</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">f</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">{</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">k</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">}</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> {</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">s[</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">'total'</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">] </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">*</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> b </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">/</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 1e9</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">:.1f</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">}</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> GB\"</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> for</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> k, b </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">in</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> BYTES</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">.items()))</span></span>\n<span class=\"line\"><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">    print</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">f</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"  FLOPs per token (matmuls, no attention scores): </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">{2</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> *</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> s[</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">'active_matmul'</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">] </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">/</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 1e9</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">:.1f</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">}</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> GFLOP\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">)</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">    kv </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> s[</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"kv_bytes_per_token\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">]</span></span>\n<span class=\"line\"><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">    print</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">f</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"  KV cache: </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">{</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">kv</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">:,</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">}</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> bytes per token, </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">{</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">kv </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">*</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 8192</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> /</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 1e9</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">:.2f</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">}</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> GB for one 8K-token sequence\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">)</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">if</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> __name__</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> ==</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> \"__main__\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">:</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    for</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> p </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">in</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> sys.argv[</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">1</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">:]:</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">        report(p)</span></span></code></pre>\n<pre class=\"shiki shiki-themes github-light github-dark\" style=\"background-color:#fff;--shiki-dark-bg:#24292e;color:#24292e;--shiki-dark:#e1e4e8\" tabindex=\"0\"><code><span class=\"line\"><span>Llama-3.2-1B-Instruct.json: 16 layers, d=2048, 32 heads / 8 KV heads of 64</span></span>\n<span class=\"line\"><span>  parameters: 1.24B stored, 1.24B in matmuls per token</span></span>\n<span class=\"line\"><span>  weights: BF16 2.5 GB, FP8 1.2 GB, INT4 0.6 GB</span></span>\n<span class=\"line\"><span>  FLOPs per token (matmuls, no attention scores): 2.5 GFLOP</span></span>\n<span class=\"line\"><span>  KV cache: 32,768 bytes per token, 0.27 GB for one 8K-token sequence</span></span>\n<span class=\"line\"><span>Meta-Llama-3.1-8B-Instruct.json: 32 layers, d=4096, 32 heads / 8 KV heads of 128</span></span>\n<span class=\"line\"><span>  parameters: 8.03B stored, 7.50B in matmuls per token</span></span>\n<span class=\"line\"><span>  weights: BF16 16.1 GB, FP8 8.0 GB, INT4 4.0 GB</span></span>\n<span class=\"line\"><span>  FLOPs per token (matmuls, no attention scores): 15.0 GFLOP</span></span>\n<span class=\"line\"><span>  KV cache: 131,072 bytes per token, 1.07 GB for one 8K-token sequence</span></span>\n<span class=\"line\"><span>Qwen3-30B-A3B.json: 48 layers, d=2048, 32 heads / 4 KV heads of 128</span></span>\n<span class=\"line\"><span>  parameters: 30.53B stored, 3.04B in matmuls per token</span></span>\n<span class=\"line\"><span>  weights: BF16 61.1 GB, FP8 30.5 GB, INT4 15.3 GB</span></span>\n<span class=\"line\"><span>  FLOPs per token (matmuls, no attention scores): 6.1 GFLOP</span></span>\n<span class=\"line\"><span>  KV cache: 98,304 bytes per token, 0.81 GB for one 8K-token sequence</span></span></code></pre>\n<p>All three totals are within 1% of the model cards (Llama 3.2 1B's card says 1.23B, which rounds the same 1.236B down). The solution handles three details that are easy to miss: <code>head_dim</code> is not always d ÷ H (Qwen3-30B-A3B has d = 2,048 but 32 heads of 128), tied embeddings are stored once, and Qwen3 adds a small norm on each query and key head.</p>\n</details>","headings":[{"id":"lab-a-model-calculator","title":"Lab: a model calculator","depth":3}]},{"id":"interview-angle","title":"Interview angle","html":"<h3 id=\"interview-angle\">Interview angle</h3>\n<ul>\n<li><strong>\"How much GPU memory do you need to serve model X to N users at context C?\"</strong> Add the weights (parameters × bytes), the KV cache (N × C × bytes per token, from the config) and about 10% for activations and the runtime. Then divide by GPU memory and round up to a GPU count the parallelism allows. State the precision you assumed.</li>\n<li><strong>\"Why does this MoE model need more GPUs than a dense model with the same per-token compute?\"</strong> Memory holds every expert, even though each token uses a few. Compute follows active parameters, memory follows total parameters.</li>\n<li><strong>\"Why do newer models have so few KV heads?\"</strong> Grouped-query attention shares each key and value head across several query heads. It shrinks the cache, and the bytes decode reads per sequence, by the group size, with little quality loss according to the GQA paper.</li>\n</ul>","headings":[{"id":"interview-angle","title":"Interview angle","depth":3}]},{"id":"check-yourself","title":"Check yourself","html":"<h3 id=\"check-yourself\">Check yourself</h3>\n<details>\n<summary>Llama 3.1 8B has 8.03B parameters, but a token's matmuls use 7.50B. Where do the other 0.53B go?</summary>\n<p>They are the input embedding table, V × d = 128,256 × 4,096 ≈ 525M parameters. Looking up a token's row is a copy, not a matrix multiply, so it costs memory but almost no compute. The LM head at the output is a separate matrix of the same size, and it is a real matmul, so it counts.</p>\n</details>\n<details>\n<summary>Work out the KV cache for one 32K-token conversation on Llama 3.1 8B in BF16, and how many such conversations fit on one H100 beside the weights.</summary>\n<p>131,072 bytes × 32,768 tokens ≈ 4.3 GB per conversation. The H100 leaves about 72 − 16.1 = 55.9 GB for cache, so about 13 conversations fit.</p>\n</details>\n<details>\n<summary>Qwen3-30B-A3B costs about 6 GFLOP per token and Llama 3.1 8B about 15. Why can the 8B model still be the cheaper one to serve?</summary>\n<p>The MoE model must hold all 30.5B parameters in memory: 61 GB in BF16, so it needs an H100-class GPU, or quantization, where the 8B fits an L4. Every decode step also reads the experts that the batch's tokens choose, and with large batches that approaches all of them. Fewer FLOPs per token help only if the memory and bandwidth costs do not dominate.</p>\n</details>\n<details>\n<summary>Why does the INT4 column say \"and up\"?</summary>\n<p>Four-bit formats store extra numbers: a scale, sometimes a zero point, per small group of weights. One 16-bit scale per 128 weights adds 0.125 bits per weight, and tools often keep the embedding and output layers in 16 bits. The real checkpoint is larger than half a byte per parameter.</p>\n</details>\n<details>\n<summary>Which is relatively worse for decode compared with prefill, the L4 or the H100? Use the ridge points.</summary>\n<p>The L4 is relatively worse. Its ridge point is about 403 FLOPs per byte against the H100's 295, so it has less bandwidth for each FLOP it can do. Decode is limited by bandwidth: the L4 has 9% of the H100's bandwidth but 12% of its compute, so decode suffers more than prefill on it.</p>\n</details>\n<h3 id=\"further-reading\">Further reading</h3>\n<ul>\n<li>The model configs and cards on Hugging Face give the shapes used here and the published parameter counts: <a href=\"https://huggingface.co/unsloth/Llama-3.2-1B-Instruct/blob/main/config.json\" target=\"_blank\" rel=\"noopener noreferrer\">Llama 3.2 1B Instruct (mirror)</a>, <a href=\"https://huggingface.co/unsloth/Meta-Llama-3.1-8B-Instruct/blob/main/config.json\" target=\"_blank\" rel=\"noopener noreferrer\">Llama 3.1 8B Instruct (mirror)</a>, <a href=\"https://huggingface.co/meta-llama/Llama-3.2-1B\" target=\"_blank\" rel=\"noopener noreferrer\">Meta's Llama 3.2 1B card</a> and <a href=\"https://huggingface.co/Qwen/Qwen3-30B-A3B\" target=\"_blank\" rel=\"noopener noreferrer\">Qwen3-30B-A3B</a>.</li>\n<li><a href=\"https://arxiv.org/abs/2407.21783\" target=\"_blank\" rel=\"noopener noreferrer\">The Llama 3 paper</a> describes the architecture and explains why it uses grouped-query attention.</li>\n<li><a href=\"https://arxiv.org/abs/2305.13245\" target=\"_blank\" rel=\"noopener noreferrer\">GQA: training generalized multi-query transformer models</a> introduces grouped-query attention and compares its quality with multi-head attention.</li>\n<li>NVIDIA's <a href=\"https://www.nvidia.com/en-us/data-center/l4/\" target=\"_blank\" rel=\"noopener noreferrer\">L4</a> and <a href=\"https://www.nvidia.com/en-us/data-center/h100/\" target=\"_blank\" rel=\"noopener noreferrer\">H100</a> product pages list memory, bandwidth and the sparsity footnotes.</li>\n<li><a href=\"https://www.apple.com/newsroom/2022/06/apple-unveils-m2-with-breakthrough-performance-and-capabilities/\" target=\"_blank\" rel=\"noopener noreferrer\">Apple's M2 announcement</a> gives the 100 GB/s unified memory bandwidth.</li>\n<li>The <a href=\"https://docs.pytorch.org/docs/stable/mps.html\" target=\"_blank\" rel=\"noopener noreferrer\">PyTorch MPS backend</a> documentation covers <code>recommended_max_memory</code> and the other MPS memory functions.</li>\n</ul>","headings":[{"id":"check-yourself","title":"Check yourself","depth":3},{"id":"further-reading","title":"Further reading","depth":3}]}]}