{"id":"training-infrastructure","title":"Training and fine-tuning infrastructure","sections":[{"id":"overview","title":"Overview","html":"<p>Serving handles requests as they arrive. Training is a batch job that runs for weeks, spreads across thousands of GPUs and has to survive hardware failures. The Llama 3 paper's flagship model has 405 billion trainable parameters, trained on 15.6 trillion tokens with 3.8 × 10^25 floating-point operations (FLOPs) on up to 16K H100 GPUs (arXiv 2407.21783). A FLOP is one floating-point addition or multiplication. A token is one piece of text, as <a href=\"/learn/llm-inference#tokens\">how LLM inference works</a> explains.</p>\n<p>Many teams fine-tune someone else's model instead. The questions are the same at every size: how much memory each GPU must hold, how much compute the job needs, how the work is split, what happens when a GPU fails, and where the data comes from.</p>\n<p>The examples use the running example's Llama 3.1 8B and 70B models on 80 GB H100 GPUs. Two estimates are worked in full: the memory to train a 70B model with Adam, and the compute to train an 8B model on 15 trillion tokens. Hardware is covered in <a href=\"/learn/gpus-for-ai\">GPUs and accelerators</a>, and serving in <a href=\"/learn/inference-engines\">inference engines</a> and <a href=\"/learn/inference-fleets\">inference fleets</a>.</p>","headings":[]},{"id":"training-compared-with-inference","title":"Training compared with inference","html":"<h3 id=\"training-compared-with-inference\">Training compared with inference</h3>\n<p>An inference step reads the weights and never changes them. For the 70B model in BF16 (16-bit floating point), a decode step at batch size one reads all 140 GB of weights.</p>\n<p>A training step runs a forward pass to measure the loss, a backward pass to compute a gradient for every weight, and an optimizer step that changes every weight using those gradients. Training is also synchronous: every data-parallel GPU must finish a step before the group can start the next, so one slow or failed GPU holds back all the others. It stores state that inference never touches, one gradient per weight plus the optimizer's running averages.</p>\n<h4 id=\"compute-per-token\">Compute per token</h4>\n<p>A forward pass costs about 2 FLOPs per parameter per token. The factor 2 comes from the multiply and the add in each matrix product (Kaplan et al., arXiv 2001.08361). The backward pass costs about twice that, so a training token costs about 6 FLOPs per parameter:</p>\n<pre class=\"shiki shiki-themes github-light github-dark\" style=\"background-color:#fff;--shiki-dark-bg:#24292e;color:#24292e;--shiki-dark:#e1e4e8\" tabindex=\"0\"><code><span class=\"line\"><span>training FLOPs ≈ 6 × parameters × tokens</span></span></code></pre>\n<p>For the 70B model, one training token costs 6 × 70 × 10^9 = 420 GFLOP.</p>\n<h4 id=\"a-worked-estimate-8b-on-15-trillion-tokens\">A worked estimate: 8B on 15 trillion tokens</h4>\n<p>The Llama 3 model card says the pre-training data was \"over 15 trillion tokens\", so this estimate is a floor.</p>\n<ol>\n<li>Total work: 6 × 8 × 10^9 × 15 × 10^12 = 7.2 × 10^23 FLOPs.</li>\n<li>Peak speed: NVIDIA's H100 page lists 1,979 teraFLOPS for BF16 Tensor Cores, marked \"with sparsity\". Sparsity doubles throughput, so the dense figure is 989 teraFLOPS (NVIDIA's Hopper architecture post).</li>\n<li>Utilization: assume 40 percent. The Llama 3 paper reports a model FLOPs utilization (MFU) of 38 to 43 percent for its 405B run. MFU is the share of peak arithmetic that goes to the model's own math. Each GPU delivers 0.40 × 989 × 10^12 = 3.956 × 10^14 FLOP/s.</li>\n<li>Time: 7.2 × 10^23 ÷ 3.956 × 10^14 = 1.82 × 10^9 GPU-seconds, or 505,561 GPU-hours and 21,065 GPU-days. On 1,000 GPUs, that is about 21 days, if nothing fails.</li>\n</ol>\n<p>The Llama 3 model card reports 1.3 million H100-80GB GPU-hours for 8B and 6.4 million for 70B. The 8B figure is 2.6 times the estimate, and the 70B figure is 1.45 times the 4.42 million estimate. The 405B case fits better. The paper's 3.8 × 10^25 FLOPs matches 6 × 405 × 10^9 × 15.6 × 10^12 = 3.79 × 10^25. The Llama 3.1 card's 30.84 million GPU-hours is 10 to 25 percent above the estimate at the paper's 38 to 43 percent MFU.</p>\n<p>The gap grows as the model shrinks, and the sources don't explain it. A plausible cause is that a small model does less math per byte moved between GPUs, so its GPUs idle more. Restarts, checkpoints and data stalls add GPU-hours that the estimate leaves out, so treat 40 percent as optimistic. The 8B run also uses 1,875 tokens per parameter, about 94 times the ratio of 20 that Chinchilla found compute-optimal (arXiv 2203.15556).</p>\n<p>At an assumed $3.00 per H100-hour (real prices vary by provider and commitment), the ideal run costs about $1.5 million, and the reported 1.3 million GPU-hours cost about $3.9 million.</p>\n<p>The sketch reproduces the arithmetic.</p>\n<pre class=\"shiki shiki-themes github-light github-dark\" style=\"background-color:#fff;--shiki-dark-bg:#24292e;color:#24292e;--shiki-dark:#e1e4e8\" tabindex=\"0\"><code><span class=\"line\"><span style=\"color:#6A737D;--shiki-dark:#6A737D\"># Training compute: 6 x parameters x tokens, converted to H100 GPU-hours at a stated MFU.</span></span>\n<span class=\"line\"><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">PEAK_BF16</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> =</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 989e12</span><span style=\"color:#6A737D;--shiki-dark:#6A737D\">  # H100 SXM dense BF16 FLOP/s (the spec's 1,979 TFLOPS is with sparsity)</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">def</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\"> training_flops</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(params: </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">float</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, tokens: </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">float</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">) -> </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">float</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">:</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    return</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 6</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> *</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> params </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">*</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> tokens</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">def</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\"> gpu_hours</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(flops: </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">float</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, mfu: </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">float</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">) -> </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">float</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">:</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    return</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> flops </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">/</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> (mfu </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">*</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> PEAK_BF16</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">) </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">/</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 3600</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">if</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> __name__</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> ==</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> \"__main__\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">:</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">    flops_8b </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> training_flops(</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">8e9</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">15e12</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">)</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">    hours_8b </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> gpu_hours(flops_8b, </span><span style=\"color:#E36209;--shiki-dark:#FFAB70\">mfu</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">0.40</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">)</span></span>\n<span class=\"line\"><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">    print</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">f</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"8B on 15T tokens: </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">{</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">flops_8b</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">:.2e</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">}</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> FLOPs, </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">{</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">hours_8b</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">:,.0f</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">}</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> GPU-hours, \"</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">          f</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">{</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">hours_8b </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">/</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 24</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">:,.0f</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">}</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> GPU-days\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">)</span></span>\n<span class=\"line\"><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">    print</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">f</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"  Llama 3 model card (1.3M GPU-hours) = </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">{1.3e6</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> /</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> hours_8b</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">:.2f</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">}</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> x this\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">)</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">    hours_70b </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> gpu_hours(training_flops(</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">70e9</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">15e12</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">), </span><span style=\"color:#E36209;--shiki-dark:#FFAB70\">mfu</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">0.40</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">)</span></span>\n<span class=\"line\"><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">    print</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">f</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"70B on 15T tokens: </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">{</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">hours_70b</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">:,.0f</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">}</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> GPU-hours; \"</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">          f</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"model card (6.4M) = </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">{6.4e6</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> /</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> hours_70b</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">:.2f</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">}</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> x this\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">)</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">    flops_405b </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> training_flops(</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">405e9</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">15.6e12</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">)</span></span>\n<span class=\"line\"><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">    print</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">f</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"405B on 15.6T tokens: </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">{</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">flops_405b</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">:.2e</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">}</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> FLOPs (paper: 3.8e25)\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">)</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    for</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> mfu </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">in</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> (</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">0.43</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">0.38</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">):</span></span>\n<span class=\"line\"><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">        print</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">f</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"  405B at MFU </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">{</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">mfu</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">:.0%</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">}</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">: </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">{</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">gpu_hours(flops_405b, mfu) </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">/</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 1e6</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">:.2f</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">}</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">M GPU-hours\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">)</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#6A737D;--shiki-dark:#6A737D\"># Output:</span></span>\n<span class=\"line\"><span style=\"color:#6A737D;--shiki-dark:#6A737D\"># 8B on 15T tokens: 7.20e+23 FLOPs, 505,561 GPU-hours, 21,065 GPU-days</span></span>\n<span class=\"line\"><span style=\"color:#6A737D;--shiki-dark:#6A737D\">#   Llama 3 model card (1.3M GPU-hours) = 2.57 x this</span></span>\n<span class=\"line\"><span style=\"color:#6A737D;--shiki-dark:#6A737D\"># 70B on 15T tokens: 4,423,660 GPU-hours; model card (6.4M) = 1.45 x this</span></span>\n<span class=\"line\"><span style=\"color:#6A737D;--shiki-dark:#6A737D\"># 405B on 15.6T tokens: 3.79e+25 FLOPs (paper: 3.8e25)</span></span>\n<span class=\"line\"><span style=\"color:#6A737D;--shiki-dark:#6A737D\">#   405B at MFU 43%: 24.76M GPU-hours</span></span>\n<span class=\"line\"><span style=\"color:#6A737D;--shiki-dark:#6A737D\">#   405B at MFU 38%: 28.02M GPU-hours</span></span></code></pre>","headings":[{"id":"training-compared-with-inference","title":"Training compared with inference","depth":3},{"id":"compute-per-token","title":"Compute per token","depth":4},{"id":"a-worked-estimate-8b-on-15-trillion-tokens","title":"A worked estimate: 8B on 15 trillion tokens","depth":4}]},{"id":"memory-for-training","title":"Memory for training","html":"<h3 id=\"memory-for-training\">Memory for training</h3>\n<p>Training here uses mixed precision with Adam, the setup behind the ZeRO paper's accounting (arXiv 1910.02054). The paper counts 16 bytes per parameter:</p>\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n<div class=\"table-wrap\"><table><thead><tr><th>What is stored</th><th>Precision</th><th>Bytes per parameter</th></tr></thead><tbody><tr><td>Weights for the forward and backward passes</td><td>16-bit</td><td>2</td></tr><tr><td>Gradients</td><td>16-bit</td><td>2</td></tr><tr><td>Master copy of the weights</td><td>32-bit</td><td>4</td></tr><tr><td>Adam momentum</td><td>32-bit</td><td>4</td></tr><tr><td>Adam variance</td><td>32-bit</td><td>4</td></tr><tr><td>Total</td><td></td><td>16</td></tr></tbody></table></div>\n<p>The paper writes this as 2Ψ + 2Ψ + KΨ = 16Ψ bytes, where Ψ is the parameter count and K = 12 is the optimizer's three 4-byte terms. These bytes are model state. Activations (values kept for the backward pass), buffers and fragmented memory come on top. The paper calls all of that residual memory and handles it separately (ZeRO-R).</p>\n<p>For the 70B model, 70 × 10^9 × 16 bytes = 1,120 GB. One H100 holds 80 GB of HBM3, the stacked memory next to the chip, so the state needs at least 1,120 ÷ 80 = 14 GPUs before activations. An H200 holds 141 GB, according to NVIDIA's H200 page, which marks the figure preliminary, so eight are the minimum. The 140 GB of BF16 weights is one eighth of the total. The 8B model needs 128 GB, which is more than one H100.</p>\n<p>So for a 70B model, the question is how many GPUs share the state, and how.</p>","headings":[{"id":"memory-for-training","title":"Memory for training","depth":3}]},{"id":"data-parallelism-and-sharding","title":"Data parallelism and sharding","html":"<h3 id=\"data-parallelism-and-sharding\">Data parallelism and sharding</h3>\n<p>In plain data parallelism, every GPU holds a full copy of the model and trains on its own slice of the batch. The GPUs average their gradients with an all-reduce, a collective operation that leaves every GPU with the same result. This adds throughput, but each GPU still holds 1,120 GB at any GPU count. FSDP calls this mode NO_SHARD.</p>\n<p>ZeRO removes the copies in three stages. With K = 12 and N data-parallel GPUs, the per-GPU model-state memory is:</p>\n<ul>\n<li>Stage 1 partitions the optimizer states: 4Ψ + KΨ/N bytes.</li>\n<li>Stage 2 also partitions the gradients: 2Ψ + 14Ψ/N bytes.</li>\n<li>Stage 3 also partitions the weights: 16Ψ/N bytes.</li>\n</ul>\n<p>The paper's example is a 7.5-billion-parameter model on 64 GPUs. It needs 120 GB per GPU with standard data parallelism, 31.4 GB at stage 1, 16.6 GB at stage 2 and 1.9 GB at stage 3 (arXiv 1910.02054). For the 70B model on 64 GPUs, the same formulas give 293 GB, 155 GB and 17.5 GB. Only stage 3 leaves room for activations on an 80 GB card.</p>\n<p>PyTorch's Fully Sharded Data Parallel (FSDP) offers these strategies. As of October 2026, its docs (version 2.14, updated May 2026) describe:</p>\n<ul>\n<li>FULL_SHARD, the default, shards parameters, gradients and optimizer states. Parameters are all-gathered (every GPU collects the missing shards) before each unit computes, then resharded. The docs say it is inspired by ZeRO stage 3.</li>\n<li>SHARD_GRAD_OP shards gradients and optimizer states, and keeps the parameters unsharded from forward through backward.</li>\n<li>NO_SHARD replicates everything, like DistributedDataParallel.</li>\n<li>HYBRID_SHARD applies FULL_SHARD within a node and replicates parameters across nodes.</li>\n</ul>\n<p>Sharding costs communication. ZeRO stage 3 moves 3Ψ bytes per step, against 2Ψ for standard data parallelism, which is 1.5 times as much (arXiv 1910.02054). HYBRID_SHARD keeps the all-gathers inside one server, where GPUs share NVLink, the fast link between GPUs in a server.</p>\n<figure class=\"diagram\" data-text=\"GPU 0 shard 1 of 4 -> All-gather layer 12; GPU 1 shard 2 of 4 -> All-gather layer 12; GPU 2 shard 3 of 4 -> All-gather layer 12; GPU 3 shard 4 of 4 -> All-gather layer 12; All-gather layer 12 -> Each GPU holds layer 12 in full (run layer 12)\"><div class=\"diagram-scroll\"><svg class=\"dg\" viewBox=\"0 0 640 320\" width=\"640\" role=\"img\" aria-label=\"Full sharding. Each layer is gathered just before it runs, then freed.\"><defs><marker id=\"dg-arrow\" viewBox=\"0 0 10 10\" refX=\"9\" refY=\"5\" markerWidth=\"7\" markerHeight=\"7\" orient=\"auto-start-reverse\"><path class=\"dg-arrowhead\" d=\"M0,1 L9,5 L0,9 z\"></path></marker></defs><line class=\"dg-edge\" x1=\"138\" y1=\"71\" x2=\"258\" y2=\"131\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge\" x1=\"259.3\" y1=\"71\" x2=\"299.3\" y2=\"131\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge\" x1=\"380.7\" y1=\"71\" x2=\"340.7\" y2=\"131\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge\" x1=\"502\" y1=\"71\" x2=\"382\" y2=\"131\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge\" x1=\"320\" y1=\"191\" x2=\"320\" y2=\"251\" marker-end=\"url(#dg-arrow)\"></line><rect class=\"dg-node dg-box\" x=\"12\" y=\"16\" width=\"136\" height=\"52\" rx=\"8\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"80\" y=\"38.5\">GPU 0</tspan><tspan x=\"80\" y=\"54.5\">shard 1 of 4</tspan></text><rect class=\"dg-node dg-box\" x=\"172\" y=\"16\" width=\"136\" height=\"52\" rx=\"8\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"240\" y=\"38.5\">GPU 1</tspan><tspan x=\"240\" y=\"54.5\">shard 2 of 4</tspan></text><rect class=\"dg-node dg-box\" x=\"332\" y=\"16\" width=\"136\" height=\"52\" rx=\"8\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"400\" y=\"38.5\">GPU 2</tspan><tspan x=\"400\" y=\"54.5\">shard 3 of 4</tspan></text><rect class=\"dg-node dg-box\" x=\"492\" y=\"16\" width=\"136\" height=\"52\" rx=\"8\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"560\" y=\"38.5\">GPU 3</tspan><tspan x=\"560\" y=\"54.5\">shard 4 of 4</tspan></text><rect class=\"dg-node dg-box\" x=\"252\" y=\"136\" width=\"136\" height=\"52\" rx=\"8\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"320\" y=\"158.5\">All-gather</tspan><tspan x=\"320\" y=\"174.5\">layer 12</tspan></text><rect class=\"dg-node dg-box\" x=\"252\" y=\"256\" width=\"136\" height=\"52\" rx=\"8\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"320\" y=\"278.5\">Each GPU holds</tspan><tspan x=\"320\" y=\"294.5\">layer 12 in full</tspan></text><rect class=\"dg-edge-bg\" x=\"277.8\" y=\"212\" width=\"84.4\" height=\"18\" rx=\"4\"></rect><text class=\"dg-edge-label\" x=\"320\" y=\"225\" text-anchor=\"middle\">run layer 12</text></svg></div><figcaption>Full sharding. Each layer is gathered just before it runs, then freed.</figcaption></figure>\n<p>The sketch below computes model-state memory per GPU with and without sharding.</p>\n<pre class=\"shiki shiki-themes github-light github-dark\" style=\"background-color:#fff;--shiki-dark-bg:#24292e;color:#24292e;--shiki-dark:#e1e4e8\" tabindex=\"0\"><code><span class=\"line\"><span style=\"color:#6A737D;--shiki-dark:#6A737D\"># Training memory per GPU: model state only (weights, gradients, Adam states).</span></span>\n<span class=\"line\"><span style=\"color:#6A737D;--shiki-dark:#6A737D\"># Mixed precision with Adam costs 16 bytes per parameter (ZeRO paper: 2P + 2P + 12P).</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">BYTES_PER_PARAM</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> =</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> {</span></span>\n<span class=\"line\"><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">    \"BF16 weights\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">2</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">,</span></span>\n<span class=\"line\"><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">    \"BF16 gradients\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">2</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">,</span></span>\n<span class=\"line\"><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">    \"FP32 master weights\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">4</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">,</span></span>\n<span class=\"line\"><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">    \"Adam momentum\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">4</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">,</span></span>\n<span class=\"line\"><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">    \"Adam variance\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">4</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">,</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">}</span></span>\n<span class=\"line\"><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">TOTAL_BYTES</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> =</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> sum</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">BYTES_PER_PARAM</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">.values())  </span><span style=\"color:#6A737D;--shiki-dark:#6A737D\"># 16</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">def</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\"> training_memory_gb</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(params: </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">float</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, gpus: </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">int</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, sharded: </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">bool</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">) -> </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">float</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">:</span></span>\n<span class=\"line\"><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">    \"\"\"GB of model state per GPU, before activations and buffers.</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">    sharded=True:  full sharding (ZeRO-3 / FSDP FULL_SHARD); each GPU holds 1/gpus.</span></span>\n<span class=\"line\"><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">    sharded=False: no sharding (plain data parallelism); each GPU holds a full copy.</span></span>\n<span class=\"line\"><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">    \"\"\"</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">    total_gb </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> params </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">*</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> TOTAL_BYTES</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> /</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 1e9</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    return</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> total_gb </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">/</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> gpus </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">if</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> sharded </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">else</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> total_gb</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">if</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> __name__</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> ==</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> \"__main__\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">:</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    for</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> name, params </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">in</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> [(</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"8B\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">8e9</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">), (</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"70B\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">70e9</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">)]:</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">        for</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> gpus </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">in</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> (</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">1</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">8</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">16</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">64</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">):</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">            none </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> training_memory_gb(params, gpus, </span><span style=\"color:#E36209;--shiki-dark:#FFAB70\">sharded</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">False</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">)</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">            full </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> training_memory_gb(params, gpus, </span><span style=\"color:#E36209;--shiki-dark:#FFAB70\">sharded</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">True</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">)</span></span>\n<span class=\"line\"><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">            print</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">f</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">{</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">name</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">:>3</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">}</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> on </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">{</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">gpus</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">:>2</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">}</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> GPUs: no sharding </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">{</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">none</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">:7.1f</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">}</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> GB, \"</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">                  f</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"full sharding </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">{</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">full</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">:6.1f</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">}</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> GB\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">)</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#6A737D;--shiki-dark:#6A737D\"># Output:</span></span>\n<span class=\"line\"><span style=\"color:#6A737D;--shiki-dark:#6A737D\">#  8B on  1 GPUs: no sharding   128.0 GB, full sharding  128.0 GB</span></span>\n<span class=\"line\"><span style=\"color:#6A737D;--shiki-dark:#6A737D\">#  8B on  8 GPUs: no sharding   128.0 GB, full sharding   16.0 GB</span></span>\n<span class=\"line\"><span style=\"color:#6A737D;--shiki-dark:#6A737D\">#  8B on 16 GPUs: no sharding   128.0 GB, full sharding    8.0 GB</span></span>\n<span class=\"line\"><span style=\"color:#6A737D;--shiki-dark:#6A737D\">#  8B on 64 GPUs: no sharding   128.0 GB, full sharding    2.0 GB</span></span>\n<span class=\"line\"><span style=\"color:#6A737D;--shiki-dark:#6A737D\"># 70B on  1 GPUs: no sharding  1120.0 GB, full sharding 1120.0 GB</span></span>\n<span class=\"line\"><span style=\"color:#6A737D;--shiki-dark:#6A737D\"># 70B on  8 GPUs: no sharding  1120.0 GB, full sharding  140.0 GB</span></span>\n<span class=\"line\"><span style=\"color:#6A737D;--shiki-dark:#6A737D\"># 70B on 16 GPUs: no sharding  1120.0 GB, full sharding   70.0 GB</span></span>\n<span class=\"line\"><span style=\"color:#6A737D;--shiki-dark:#6A737D\"># 70B on 64 GPUs: no sharding  1120.0 GB, full sharding   17.5 GB</span></span></code></pre>\n<p>At 8 GPUs, full sharding leaves 140 GB per GPU for the 70B model, still too big for an 80 GB card. At 16 GPUs it leaves 70 GB, so about 10 GB remains for activations. These are model-state numbers only, so they set a floor on the GPU count.</p>","headings":[{"id":"data-parallelism-and-sharding","title":"Data parallelism and sharding","depth":3}]},{"id":"tensor-pipeline-and-3d-parallelism","title":"Tensor, pipeline and 3D parallelism","html":"<h3 id=\"tensor-pipeline-and-3d-parallelism\">Tensor, pipeline and 3D parallelism</h3>\n<p>Sharding fixes memory, but each GPU still runs the full math of every layer. When one layer's math is too big for one GPU, or the layers are too many for one server, the math must be split.</p>\n<p>Tensor parallelism splits the matrix products inside each layer. The 2019 Megatron-LM paper splits the MLP's first matrix by columns, so each GPU computes its own part of the hidden layer, and the second by rows, followed by one all-reduce. Attention heads split the same way. Each layer needs two all-reduces in the forward pass and two in the backward pass, on every layer's critical path, so tensor parallelism needs fast links. The 2021 paper says it works best inside one server and can be impractical across the slower links between servers (arXiv 1909.08053, arXiv 2104.04473). The Llama 3 16K layout keeps tensor-parallel size at 8, the GPUs in one server (arXiv 2407.21783, section 3.3 and Table 4).</p>\n<p>Pipeline parallelism splits the layers into stages on different servers and cuts each batch into microbatches that flow through them. The cost is the bubble, the idle time at the start and end of each batch. The 2021 Megatron-LM paper gives the lost fraction as (p − 1) ÷ m, where p is the number of stages and m is the number of microbatches (arXiv 2104.04473). With 4 stages and 8 microbatches, that is 3 ÷ 8 = 37.5 percent. With 32 microbatches, it is 3 ÷ 32 = 9.4 percent. Interleaving gives each device v smaller groups of layers, which divides the bubble by v at the cost of more communication.</p>\n<figure class=\"diagram\" data-text=\"R1 stage 1 layers 1-20 -> R1 stage 2 layers 21-40 (activations); R1 stage 2 layers 21-40 -> R1 stage 3 layers 41-60 (activations); R1 stage 3 layers 41-60 -> R1 stage 4 layers 61-80 (activations); R2 stage 1 layers 1-20 -> R2 stage 2 layers 21-40 (activations); R2 stage 2 layers 21-40 -> R2 stage 3 layers 41-60 (activations); R2 stage 3 layers 41-60 -> R2 stage 4 layers 61-80 (activations); R1 stage 1 layers 1-20 <-> R2 stage 1 layers 1-20 (gradients all-reduce)\"><div class=\"diagram-scroll\"><svg class=\"dg\" viewBox=\"0 -10 640 330\" width=\"640\" role=\"img\" aria-label=\"Two replicas of a 4-stage pipeline. Each stage is one server of 8 GPUs.\"><defs><marker id=\"dg-arrow\" viewBox=\"0 0 10 10\" refX=\"9\" refY=\"5\" markerWidth=\"7\" markerHeight=\"7\" orient=\"auto-start-reverse\"><path class=\"dg-arrowhead\" d=\"M0,1 L9,5 L0,9 z\"></path></marker></defs><line class=\"dg-edge\" x1=\"151\" y1=\"42\" x2=\"167\" y2=\"42\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge\" x1=\"311\" y1=\"42\" x2=\"327\" y2=\"42\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge\" x1=\"471\" y1=\"42\" x2=\"487\" y2=\"42\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge\" x1=\"151\" y1=\"282\" x2=\"167\" y2=\"282\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge\" x1=\"311\" y1=\"282\" x2=\"327\" y2=\"282\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge\" x1=\"471\" y1=\"282\" x2=\"487\" y2=\"282\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge\" x1=\"80\" y1=\"73\" x2=\"80\" y2=\"251\" marker-end=\"url(#dg-arrow)\" marker-start=\"url(#dg-arrow)\"></line><rect class=\"dg-node dg-box\" x=\"12\" y=\"16\" width=\"136\" height=\"52\" rx=\"8\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"80\" y=\"38.5\">R1 stage 1</tspan><tspan x=\"80\" y=\"54.5\">layers 1-20</tspan></text><rect class=\"dg-node dg-box\" x=\"172\" y=\"16\" width=\"136\" height=\"52\" rx=\"8\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"240\" y=\"38.5\">R1 stage 2</tspan><tspan x=\"240\" y=\"54.5\">layers 21-40</tspan></text><rect class=\"dg-node dg-box\" x=\"332\" y=\"16\" width=\"136\" height=\"52\" rx=\"8\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"400\" y=\"38.5\">R1 stage 3</tspan><tspan x=\"400\" y=\"54.5\">layers 41-60</tspan></text><rect class=\"dg-node dg-box\" x=\"492\" y=\"16\" width=\"136\" height=\"52\" rx=\"8\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"560\" y=\"38.5\">R1 stage 4</tspan><tspan x=\"560\" y=\"54.5\">layers 61-80</tspan></text><rect class=\"dg-node dg-box\" x=\"12\" y=\"256\" width=\"136\" height=\"52\" rx=\"8\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"80\" y=\"278.5\">R2 stage 1</tspan><tspan x=\"80\" y=\"294.5\">layers 1-20</tspan></text><rect class=\"dg-node dg-box\" x=\"172\" y=\"256\" width=\"136\" height=\"52\" rx=\"8\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"240\" y=\"278.5\">R2 stage 2</tspan><tspan x=\"240\" y=\"294.5\">layers 21-40</tspan></text><rect class=\"dg-node dg-box\" x=\"332\" y=\"256\" width=\"136\" height=\"52\" rx=\"8\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"400\" y=\"278.5\">R2 stage 3</tspan><tspan x=\"400\" y=\"294.5\">layers 41-60</tspan></text><rect class=\"dg-node dg-box\" x=\"492\" y=\"256\" width=\"136\" height=\"52\" rx=\"8\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"560\" y=\"278.5\">R2 stage 4</tspan><tspan x=\"560\" y=\"294.5\">layers 61-80</tspan></text><rect class=\"dg-edge-bg\" x=\"119.9\" y=\"-6\" width=\"78.2\" height=\"18\" rx=\"4\"></rect><text class=\"dg-edge-label\" x=\"159\" y=\"7\" text-anchor=\"middle\">activations</text><rect class=\"dg-edge-bg\" x=\"279.9\" y=\"-6\" width=\"78.2\" height=\"18\" rx=\"4\"></rect><text class=\"dg-edge-label\" x=\"319\" y=\"7\" text-anchor=\"middle\">activations</text><rect class=\"dg-edge-bg\" x=\"439.9\" y=\"-6\" width=\"78.2\" height=\"18\" rx=\"4\"></rect><text class=\"dg-edge-label\" x=\"479\" y=\"7\" text-anchor=\"middle\">activations</text><rect class=\"dg-edge-bg\" x=\"119.9\" y=\"234\" width=\"78.2\" height=\"18\" rx=\"4\"></rect><text class=\"dg-edge-label\" x=\"159\" y=\"247\" text-anchor=\"middle\">activations</text><rect class=\"dg-edge-bg\" x=\"279.9\" y=\"234\" width=\"78.2\" height=\"18\" rx=\"4\"></rect><text class=\"dg-edge-label\" x=\"319\" y=\"247\" text-anchor=\"middle\">activations</text><rect class=\"dg-edge-bg\" x=\"439.9\" y=\"234\" width=\"78.2\" height=\"18\" rx=\"4\"></rect><text class=\"dg-edge-label\" x=\"479\" y=\"247\" text-anchor=\"middle\">activations</text><rect class=\"dg-edge-bg\" x=\"13\" y=\"153\" width=\"134\" height=\"18\" rx=\"4\"></rect><text class=\"dg-edge-label\" x=\"80\" y=\"166\" text-anchor=\"middle\">gradients all-reduce</text></svg></div><figcaption>Two replicas of a 4-stage pipeline. Each stage is one server of 8 GPUs.</figcaption></figure>\n<p>Large runs combine the kinds. The 2021 paper's 1-trillion-parameter run used 3,072 GPUs at 502 petaFLOP/s, 52 percent of peak per GPU. The Llama 3 16K layout is tensor size 8 × pipeline size 16 × data size 128 = 16,384 GPUs, with context-parallel size 1 (Table 4). It reaches 41 percent MFU, against 43 percent at 8K GPUs with data-parallel size 64. The paper attributes the drop to a smaller batch per data-parallel group, since the batch stays at 16 million tokens (section 3.3.2).</p>","headings":[{"id":"tensor-pipeline-and-3d-parallelism","title":"Tensor, pipeline and 3D parallelism","depth":3}]},{"id":"failures-and-checkpoints","title":"Failures and checkpoints","html":"<h3 id=\"failures-and-checkpoints\">Failures and checkpoints</h3>\n<p>At 16K GPUs, failures are routine. Over a 54-day pre-training snapshot, the Llama 3 paper counts 466 job interruptions: 47 planned for maintenance and 419 unexpected (arXiv 2407.21783, section 3.3.4). Table 5 lists the largest causes as faulty GPUs (148), GPU HBM3 memory (72), software bugs (54), network switches and cables (35) and host maintenance (32).</p>\n<p>Fifty-four days is 1,296 hours, so an unexpected interruption came about every 3.1 hours. If every one were a GPU failure, each GPU would fail about once every 5.8 years. Many causes aren't GPUs, so the real per-GPU rate is lower. A failure is rare for any one GPU and near certain for the job, so the job has to plan for it.</p>\n<p>A checkpoint saves the weights, optimizer states, data position and random-number state. The paper says each GPU's share ranges from 1 MB to 4 GB, and that checkpoint writes are \"highly bursty\", saturating the storage fabric for short periods (sections 3.3.1 and 3.3.4). At 16 bytes per parameter, a 405B model's state is about 6.5 TB.</p>\n<p>Checkpointing trades two costs. Checkpoint rarely, and each failure loses a long stretch of work. Checkpoint often, and training stalls each time. Daly's 2006 refinement of Young's first-order rule gives the balance (see further reading). With δ the stall per checkpoint and M the mean time between interruptions, the best interval is √(2δM) − δ, when δ is less than M/2. Assume a 2-minute stall and M = 185.6 minutes. The interval is √(2 × 2 × 185.6) − 2 = 25.2 minutes, and a failure costs about half of that, 12.6 minutes, plus restart time.</p>\n<p>Asynchronous checkpointing shrinks the stall. PyTorch's distributed checkpoint API copies the state from GPU memory to CPU memory, then writes it to storage on a background thread by default (PyTorch docs, version 2.14). Training waits only for the copy. If the copy takes an assumed 10 seconds, the interval becomes 7.7 minutes, and the work lost per failure falls to about 3.8 minutes.</p>\n<figure class=\"diagram\" data-text=\"GPUs step N -> CPU memory staged copy (copy, stalls training); CPU memory staged copy -> Storage storage fabric (background write)\"><div class=\"diagram-scroll\"><svg class=\"dg\" viewBox=\"0 -15 480 100\" width=\"480\" role=\"img\" aria-label=\"Async checkpoint. Training stalls only for the copy to CPU memory.\"><defs><marker id=\"dg-arrow\" viewBox=\"0 0 10 10\" refX=\"9\" refY=\"5\" markerWidth=\"7\" markerHeight=\"7\" orient=\"auto-start-reverse\"><path class=\"dg-arrowhead\" d=\"M0,1 L9,5 L0,9 z\"></path></marker></defs><line class=\"dg-edge\" x1=\"151\" y1=\"42\" x2=\"167\" y2=\"42\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge dg-dashed\" x1=\"311\" y1=\"42\" x2=\"327\" y2=\"42\" marker-end=\"url(#dg-arrow)\"></line><rect class=\"dg-node dg-client\" x=\"12\" y=\"16\" width=\"136\" height=\"52\" rx=\"26\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"80\" y=\"38.5\">GPUs</tspan><tspan x=\"80\" y=\"54.5\">step N</tspan></text><rect class=\"dg-node dg-box\" x=\"172\" y=\"16\" width=\"136\" height=\"52\" rx=\"8\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"240\" y=\"38.5\">CPU memory</tspan><tspan x=\"240\" y=\"54.5\">staged copy</tspan></text><path class=\"dg-node dg-store\" d=\"M332,18 a68,7 0 0,1 136,0 v48 a68,7 0 0,1 -136,0 z\"></path><path class=\"dg-line\" d=\"M332,18 a68,7 0 0,0 136,0\"></path><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"400\" y=\"41.5\">Storage</tspan><tspan x=\"400\" y=\"57.5\">storage fabric</tspan></text><rect class=\"dg-edge-bg\" x=\"88.9\" y=\"-6\" width=\"140.2\" height=\"18\" rx=\"4\"></rect><text class=\"dg-edge-label\" x=\"159\" y=\"7\" text-anchor=\"middle\">copy, stalls training</text><rect class=\"dg-edge-bg\" x=\"264.4\" y=\"-11\" width=\"109.2\" height=\"18\" rx=\"4\"></rect><text class=\"dg-edge-label\" x=\"319\" y=\"2\" text-anchor=\"middle\">background write</text></svg></div><figcaption>Async checkpoint. Training stalls only for the copy to CPU memory.</figcaption></figure>\n<pre class=\"shiki shiki-themes github-light github-dark\" style=\"background-color:#fff;--shiki-dark-bg:#24292e;color:#24292e;--shiki-dark:#e1e4e8\" tabindex=\"0\"><code><span class=\"line\"><span style=\"color:#6A737D;--shiki-dark:#6A737D\"># Daly's higher-order checkpoint interval: sqrt(2 * delta * M) - delta, valid when delta &#x3C; M / 2.</span></span>\n<span class=\"line\"><span style=\"color:#6A737D;--shiki-dark:#6A737D\"># Restart time is left out because Daly's higher-order result shows it drops out.</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">from</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> math </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">import</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> sqrt</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">def</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\"> checkpoint_interval_min</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(stall_min: </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">float</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, mtti_min: </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">float</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">) -> </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">float</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">:</span></span>\n<span class=\"line\"><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">    \"\"\"stall_min: time each checkpoint stalls training (delta).</span></span>\n<span class=\"line\"><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">    mtti_min: mean time to an unexpected interruption (M).\"\"\"</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    if</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> stall_min </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">>=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> mtti_min </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">/</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 2</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">:</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">        return</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> mtti_min</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    return</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> sqrt(</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">2</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> *</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> stall_min </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">*</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> mtti_min) </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">-</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> stall_min</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">if</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> __name__</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> ==</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> \"__main__\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">:</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">    mtti </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 54</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> *</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 24</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> *</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 60</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> /</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 419</span><span style=\"color:#6A737D;--shiki-dark:#6A737D\">  # Llama 3 snapshot: 54 days, 419 unexpected interruptions</span></span>\n<span class=\"line\"><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">    print</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">f</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"MTTI: </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">{</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">mtti</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">:.1f</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">}</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> minutes (</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">{</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">mtti </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">/</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 60</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">:.2f</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">}</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> hours)\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">)</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    for</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> label, stall </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">in</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> ((</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"2-minute stall\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">2.0</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">), (</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"10-second stall\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">10</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> /</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 60</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">)):</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">        t </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> checkpoint_interval_min(stall, mtti)</span></span>\n<span class=\"line\"><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">        print</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">f</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">{</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">label</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">}</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">: checkpoint every </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">{</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">t</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">:.1f</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">}</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> min, about </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">{</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">t </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">/</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 2</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">:.1f</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">}</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> min lost per failure\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">)</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#6A737D;--shiki-dark:#6A737D\"># Output:</span></span>\n<span class=\"line\"><span style=\"color:#6A737D;--shiki-dark:#6A737D\"># MTTI: 185.6 minutes (3.09 hours)</span></span>\n<span class=\"line\"><span style=\"color:#6A737D;--shiki-dark:#6A737D\"># 2-minute stall: checkpoint every 25.2 min, about 12.6 min lost per failure</span></span>\n<span class=\"line\"><span style=\"color:#6A737D;--shiki-dark:#6A737D\"># 10-second stall: checkpoint every 7.7 min, about 3.8 min lost per failure</span></span></code></pre>","headings":[{"id":"failures-and-checkpoints","title":"Failures and checkpoints","depth":3}]},{"id":"the-data-pipeline","title":"The data pipeline","html":"<h3 id=\"the-data-pipeline\">The data pipeline</h3>\n<p>The data pipeline collects and filters text, removes duplicates, tokenizes, and feeds each GPU its shards in a fixed order. The Llama 3 paper removes duplicates at three levels (arXiv 2407.21783, section 3.1.1). URL-level keeps the newest copy of each page. Document-level runs a global MinHash pass that finds near-duplicates without comparing every pair. Line-level drops lines that appear more than six times in each bucket of 30 million documents. The final mix is roughly 50 percent general knowledge, 25 percent math and reasoning, 17 percent code and 8 percent multilingual text (section 3.1.2).</p>\n<p>Tokenize once. Running the tokenizer on every pass wastes time, so a common design writes token IDs into binary shards with an index. Megatron-LM's preprocessing script does this, writing a .bin file of IDs and a matching .idx index from JSON lines (tools/preprocess_data.py in its repository). Each data-parallel group reads its own shards in a fixed order, and the loader's position belongs in every checkpoint. A restart that rereads data repeats documents, and one that skips ahead loses them.</p>\n<p>Size the pipeline from the compute. At 40 percent MFU, one H100 consumes 3.956 × 10^14 ÷ (6 × 8 × 10^9) ≈ 8,240 tokens per second on the 8B model, so a 2,000-GPU job needs about 16.5 million tokens per second.</p>","headings":[{"id":"the-data-pipeline","title":"The data pipeline","depth":3}]},{"id":"fine-tuning","title":"Fine-tuning","html":"<h3 id=\"fine-tuning\">Fine-tuning</h3>\n<p>Full fine-tuning updates all 70 billion weights and needs the same 1,120 GB of state. That is why LoRA and QLoRA are the practical options.</p>\n<p>Supervised fine-tuning (SFT) trains on prompts paired with the responses you want. InstructGPT's first step did this with labelers' demonstrations: \"We then fine-tune a pretrained GPT-3 model on this data using supervised learning\" (arXiv 2203.02155).</p>\n<p>LoRA freezes each weight matrix W and learns a low-rank change BA, where B is d × r and A is r × k, with r much smaller than d or k. Only A and B get gradients (arXiv 2106.09685). For a 4,096 × 4,096 matrix, the model dimension of Llama 3.1 8B, a full update has 16,777,216 weights. Rank 8 trains 8 × (4,096 + 4,096) = 65,536 of them, or 0.39 percent. The paper says rank one or two suffices for GPT-3 175B even at a full dimension of 12,288, and its abstract reports 10,000 times fewer trainable parameters and 3 times less GPU memory than full fine-tuning with Adam. The frozen base still needs its weights: 16 GB in BF16 for 8B, which leaves most of an 80 GB H100 for activations, and 140 GB for 70B. After training, BA merges into W with no added inference latency, but a merged model serves one adapter (<a href=\"/learn/inference-fleets#serving-many-models\">inference fleets</a> covers serving many).</p>\n<p>QLoRA keeps the base model in 4 bits and trains 16-bit adapters on top (arXiv 2305.14314). It adds NF4, a 4-bit data type for normally distributed weights, double quantization (about 0.37 bits saved per parameter), and paged optimizers, which move optimizer memory between CPU and GPU to avoid out-of-memory crashes. It fine-tunes a 65-billion-parameter model on one 48 GB GPU, where 16-bit tuning needs more than 780 GB. For the 70B model, the 4-bit base is 70 × 10^9 × 0.5 bytes = 35 GB. The paper also finds that adapters must sit on all linear layers to match full fine-tuning.</p>\n<p>Preference tuning comes after SFT. RLHF, the method InstructGPT used, trains a reward model on labelers' rankings, then runs PPO against it while sampling answers (arXiv 2203.02155). DPO (Direct Preference Optimization) trains on preferred and dispreferred pairs against a frozen reference, the model before tuning. It needs no reward model and no sampling, but it keeps two copies of the model in memory (arXiv 2305.18290). On summarization, DPO beat PPO-based RLHF in the paper's GPT-4-judged comparison.</p>","headings":[{"id":"fine-tuning","title":"Fine-tuning","depth":3}]},{"id":"running-a-fine-tuning-service","title":"Running a fine-tuning service","html":"<h3 id=\"running-a-fine-tuning-service\">Running a fine-tuning service</h3>\n<p>A company that fine-tunes for many teams needs a service with these parts:</p>\n<ul>\n<li><strong>Job spec.</strong> The pinned base model version, method, dataset version, hyperparameters, GPU count and owner. An adapter trained on one base version can't be assumed to work on another.</li>\n<li><strong>Queue and quotas.</strong> Jobs wait by priority, and each team has a GPU quota, so one sweep can't take the cluster.</li>\n<li><strong>Scheduler.</strong> LoRA jobs fit on a few GPUs. A full 70B fine-tune needs at least 14 GPUs for its state, so it runs as a gang.</li>\n<li><strong>Workers.</strong> They checkpoint, and after a failure they resume from the newest complete checkpoint.</li>\n<li><strong>Evaluation gate.</strong> Publish the adapter only if it passes the same eval set as the base model (<a href=\"/learn/ai-evals-and-observability#running-evals-before-release\">AI evals and observability</a>).</li>\n<li><strong>Registry.</strong> Store the adapter as safetensors, a format built to store tensors safely, as opposed to pickle (Hugging Face's safetensors docs). Record the base, dataset, hyperparameters and eval results (<a href=\"/learn/ai-security#the-model-supply-chain\">AI security</a>).</li>\n<li><strong>Publish.</strong> The inference fleet loads the adapter next to the base model.</li>\n</ul>\n<p>A 70B LoRA run on 8 GPUs for 12 hours is 96 GPU-hours, or $288 at an assumed $3.00 per GPU-hour. A higher-priority job should preempt a lower one only after the lower job checkpoints, or the preemption throws away its work. That is why every section above assumes checkpoints.</p>\n<figure class=\"diagram\" data-text=\"Job API -> Job queue per-team quotas (submit); Job queue per-team quotas -> Scheduler gang placement; Scheduler gang placement -> Workers checkpoint and resume (start all workers); Workers checkpoint and resume -> Eval gate against base model (checkpoint); Eval gate against base model -> Adapter registry (pass); Adapter registry -> Inference fleet loads the adapter (publish)\"><div class=\"diagram-scroll\"><svg class=\"dg\" viewBox=\"0 -18 640 221\" width=\"640\" role=\"img\" aria-label=\"A fine-tuning job, from request to served adapter.\"><defs><marker id=\"dg-arrow\" viewBox=\"0 0 10 10\" refX=\"9\" refY=\"5\" markerWidth=\"7\" markerHeight=\"7\" orient=\"auto-start-reverse\"><path class=\"dg-arrowhead\" d=\"M0,1 L9,5 L0,9 z\"></path></marker></defs><line class=\"dg-edge\" x1=\"151\" y1=\"42\" x2=\"167\" y2=\"42\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge\" x1=\"311\" y1=\"42\" x2=\"327\" y2=\"42\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge\" x1=\"471\" y1=\"42\" x2=\"487\" y2=\"42\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge\" x1=\"560\" y1=\"79\" x2=\"560\" y2=\"131\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge\" x1=\"489\" y1=\"162\" x2=\"473\" y2=\"162\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge\" x1=\"329\" y1=\"162\" x2=\"313\" y2=\"162\" marker-end=\"url(#dg-arrow)\"></line><rect class=\"dg-node dg-client\" x=\"12\" y=\"18\" width=\"136\" height=\"48\" rx=\"24\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"80\" y=\"46.5\">Job API</tspan></text><rect class=\"dg-node dg-queue\" x=\"172\" y=\"16\" width=\"136\" height=\"52\" rx=\"6\"></rect><line class=\"dg-line\" x1=\"292\" y1=\"24\" x2=\"292\" y2=\"60\"></line><line class=\"dg-line\" x1=\"282\" y1=\"24\" x2=\"282\" y2=\"60\"></line><line class=\"dg-line\" x1=\"272\" y1=\"24\" x2=\"272\" y2=\"60\"></line><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"226\" y=\"38.5\">Job queue</tspan><tspan x=\"226\" y=\"54.5\">per-team quotas</tspan></text><rect class=\"dg-node dg-box\" x=\"332\" y=\"16\" width=\"136\" height=\"52\" rx=\"8\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"400\" y=\"38.5\">Scheduler</tspan><tspan x=\"400\" y=\"54.5\">gang placement</tspan></text><rect class=\"dg-node dg-box\" x=\"492\" y=\"8\" width=\"136\" height=\"68\" rx=\"8\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"560\" y=\"30.5\">Workers</tspan><tspan x=\"560\" y=\"46.5\">checkpoint and</tspan><tspan x=\"560\" y=\"62.5\">resume</tspan></text><rect class=\"dg-node dg-box\" x=\"492\" y=\"136\" width=\"136\" height=\"52\" rx=\"8\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"560\" y=\"158.5\">Eval gate</tspan><tspan x=\"560\" y=\"174.5\">against base model</tspan></text><path class=\"dg-node dg-store\" d=\"M332,140 a68,7 0 0,1 136,0 v44 a68,7 0 0,1 -136,0 z\"></path><path class=\"dg-line\" d=\"M332,140 a68,7 0 0,0 136,0\"></path><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"400\" y=\"169.5\">Adapter registry</tspan></text><rect class=\"dg-node dg-box\" x=\"172\" y=\"136\" width=\"136\" height=\"52\" rx=\"8\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"240\" y=\"158.5\">Inference fleet</tspan><tspan x=\"240\" y=\"174.5\">loads the adapter</tspan></text><rect class=\"dg-edge-bg\" x=\"135.4\" y=\"-6\" width=\"47.2\" height=\"18\" rx=\"4\"></rect><text class=\"dg-edge-label\" x=\"159\" y=\"7\" text-anchor=\"middle\">submit</text><rect class=\"dg-edge-bg\" x=\"421.3\" y=\"-14\" width=\"115.4\" height=\"18\" rx=\"4\"></rect><text class=\"dg-edge-label\" x=\"479\" y=\"-1\" text-anchor=\"middle\">start all workers</text><rect class=\"dg-edge-bg\" x=\"524\" y=\"96\" width=\"72\" height=\"18\" rx=\"4\"></rect><text class=\"dg-edge-label\" x=\"560\" y=\"109\" text-anchor=\"middle\">checkpoint</text><rect class=\"dg-edge-bg\" x=\"463.6\" y=\"111\" width=\"34.8\" height=\"18\" rx=\"4\"></rect><text class=\"dg-edge-label\" x=\"481\" y=\"124\" text-anchor=\"middle\">pass</text><rect class=\"dg-edge-bg\" x=\"294.3\" y=\"111\" width=\"53.4\" height=\"18\" rx=\"4\"></rect><text class=\"dg-edge-label\" x=\"321\" y=\"124\" text-anchor=\"middle\">publish</text></svg></div><figcaption>A fine-tuning job, from request to served adapter.</figcaption></figure>","headings":[{"id":"running-a-fine-tuning-service","title":"Running a fine-tuning service","depth":3}]},{"id":"scheduling-gpu-clusters","title":"Scheduling GPU clusters","html":"<h3 id=\"scheduling-gpu-clusters\">Scheduling GPU clusters</h3>\n<p>Training jobs run as gangs, groups of workers that must start together. A synchronous job can't progress until all its workers run, so the scheduler places all of a job's workers, or none.</p>\n<p>Without that rule, two jobs that each need 4 servers, with 6 free, could each get 3 and wait forever while holding GPUs. Kubernetes documents gang scheduling as all-or-nothing: \"If the cluster cannot accommodate the entire group (or a defined minimum number of Pods, specified by minCount), none of the Pods are bound to a node.\" As of the v1.37 docs, it is a beta feature, disabled by default (Kubernetes documentation). Kubeflow Trainer's coscheduling option makes a training job's pods start together, only when all their resources are available (Kubeflow Trainer documentation).</p>\n<p>Slurm, the scheduler common in high-performance computing, calls itself \"an open source, fault-tolerant, and highly scalable cluster management and job scheduling system\". Its documentation uses \"gang scheduling\" for time sharing of parallel jobs, a different idea, so ask which meaning someone means (Slurm documentation).</p>\n<p>Placement matters as much as counts. A scheduler that ignores topology can split a tensor-parallel group across two servers, sending its all-reduces over the slower links (arXiv 2104.04473). A 16,384-GPU job needs 2,048 servers at once, so on a busy cluster the scheduler must reserve capacity, wait, or preempt smaller jobs that checkpoint.</p>","headings":[{"id":"scheduling-gpu-clusters","title":"Scheduling GPU clusters","depth":3}]},{"id":"check-yourself","title":"Check yourself","html":"<h3 id=\"check-yourself\">Check yourself</h3>\n<details>\n<summary>A team fully fine-tunes a 13-billion-parameter model with Adam in mixed precision on 8 GPUs of 80 GB each. How much model state does each GPU hold with and without full sharding?</summary>\n<p>The state is 13 × 10^9 × 16 = 208 GB. Unsharded, each GPU holds all 208 GB. Fully sharded, each holds 208 ÷ 8 = 26 GB.</p>\n</details>\n<details>\n<summary>At 40 percent MFU on 989 teraFLOP/s H100s, how many GPU-days does 6 × parameters × tokens give for a 70B model on 2 trillion tokens? How long is that on 1,024 GPUs?</summary>\n<p>6 × 70 × 10^9 × 2 × 10^12 = 8.4 × 10^23 FLOPs, or 589,821 GPU-hours (24,576 GPU-days). On 1,024 GPUs, that is 24 days.</p>\n</details>\n<details>\n<summary>An 8-stage pipeline runs 16 microbatches. What fraction of time is the bubble, and how many microbatches keep it under 5 percent, with and without 2 interleaved chunks per device?</summary>\n<p>The bubble is (8 − 1) ÷ 16 = 43.75 percent. Under 5 percent needs at least 141 microbatches, or at least 71 with 2 chunks per device.</p>\n</details>\n<details>\n<summary>A 4,096-GPU job has an unexpected interruption every 3 hours, and each checkpoint stalls training for 90 seconds. What interval does Daly's rule suggest, and how much work does a failure cost?</summary>\n<p>The interval is √(2 × 1.5 × 180) − 1.5 ≈ 21.7 minutes. A failure costs about half of that on average, roughly 11 minutes, plus restart time.</p>\n</details>\n<details>\n<summary>Why won't a 70B model fit on one 80 GB H100 with LoRA in BF16, and what changes that?</summary>\n<p>The frozen BF16 base is 140 GB. QLoRA's 4-bit base is 35 GB and fits. Sharding the base across GPUs also works.</p>\n</details>\n<details>\n<summary>Two training jobs each need 4 servers. With 6 free servers and one-at-a-time placement, each job gets 3. What goes wrong, and what should the scheduler do?</summary>\n<p>Neither job can start, and the six servers they hold do no useful work. A gang scheduler starts one job on 4 servers and queues the other.</p>\n</details>\n<h3 id=\"further-reading\">Further reading</h3>\n<ul>\n<li><a href=\"https://arxiv.org/abs/2407.21783\" target=\"_blank\" rel=\"noopener noreferrer\">Llama 3 herd of models (arXiv 2407.21783)</a>: 405B compute, the 16K-GPU layout and the interruption table.</li>\n<li><a href=\"https://github.com/meta-llama/llama3/blob/main/MODEL_CARD.md\" target=\"_blank\" rel=\"noopener noreferrer\">Llama 3 model card</a>: 8B and 70B GPU-hours.</li>\n<li><a href=\"https://github.com/meta-llama/llama-models/blob/main/models/llama3_1/MODEL_CARD.md\" target=\"_blank\" rel=\"noopener noreferrer\">Llama 3.1 model card</a>: later GPU-hours, including 405B.</li>\n<li><a href=\"https://arxiv.org/abs/1910.02054\" target=\"_blank\" rel=\"noopener noreferrer\">ZeRO (arXiv 1910.02054)</a>: bytes per parameter and the three stages.</li>\n<li><a href=\"https://docs.pytorch.org/docs/stable/fsdp.html\" target=\"_blank\" rel=\"noopener noreferrer\">PyTorch FSDP documentation</a>: the sharding strategies.</li>\n<li><a href=\"https://arxiv.org/abs/1909.08053\" target=\"_blank\" rel=\"noopener noreferrer\">Megatron-LM, 2019 (arXiv 1909.08053)</a>: tensor parallelism inside a layer.</li>\n<li><a href=\"https://arxiv.org/abs/2104.04473\" target=\"_blank\" rel=\"noopener noreferrer\">Megatron-LM, 2021 (arXiv 2104.04473)</a>: the bubble formula, interleaving and 3D parallelism.</li>\n<li><a href=\"https://arxiv.org/abs/2106.09685\" target=\"_blank\" rel=\"noopener noreferrer\">LoRA (arXiv 2106.09685)</a>: low-rank adapters and rank results.</li>\n<li><a href=\"https://arxiv.org/abs/2305.14314\" target=\"_blank\" rel=\"noopener noreferrer\">QLoRA (arXiv 2305.14314)</a>: 4-bit base models, NF4 and paged optimizers.</li>\n<li><a href=\"https://arxiv.org/abs/2305.18290\" target=\"_blank\" rel=\"noopener noreferrer\">Direct Preference Optimization (arXiv 2305.18290)</a>: the DPO loss.</li>\n<li><a href=\"https://arxiv.org/abs/2203.02155\" target=\"_blank\" rel=\"noopener noreferrer\">InstructGPT (arXiv 2203.02155)</a>: the three-step RLHF pipeline.</li>\n<li><a href=\"https://arxiv.org/abs/2001.08361\" target=\"_blank\" rel=\"noopener noreferrer\">Kaplan et al., scaling laws (arXiv 2001.08361)</a>: where the factor 6 comes from.</li>\n<li><a href=\"https://arxiv.org/abs/2203.15556\" target=\"_blank\" rel=\"noopener noreferrer\">Chinchilla (arXiv 2203.15556)</a>: compute-optimal token counts.</li>\n<li><a href=\"https://www.nvidia.com/en-us/data-center/h100/\" target=\"_blank\" rel=\"noopener noreferrer\">NVIDIA H100 product page</a>: memory and sparsity figures.</li>\n<li><a href=\"https://www.nvidia.com/en-us/data-center/h200/\" target=\"_blank\" rel=\"noopener noreferrer\">NVIDIA H200 product page</a>: the 141 GB figure, marked preliminary.</li>\n<li><a href=\"https://developer.nvidia.com/blog/nvidia-hopper-architecture-in-depth/\" target=\"_blank\" rel=\"noopener noreferrer\">NVIDIA Hopper architecture in depth</a>: why sparsity doubles throughput.</li>\n<li><a href=\"https://docs.pytorch.org/docs/stable/distributed.checkpoint.html\" target=\"_blank\" rel=\"noopener noreferrer\">PyTorch distributed checkpoint documentation</a>: copy-then-write async saves.</li>\n<li><a href=\"https://laro.lanl.gov/esploro/outputs/journalArticle/A-higher-order-estimate-of-the/9916364420003761\" target=\"_blank\" rel=\"noopener noreferrer\">Daly, \"A higher order estimate of the optimum checkpoint interval for restart dumps\" (2006)</a>: the interval rule.</li>\n<li><a href=\"https://blog.acolyer.org/2015/06/10/a-higher-order-estimate-of-the-optimun-checkpoint-interval-for-restart-dumps/\" target=\"_blank\" rel=\"noopener noreferrer\">Blog summary of Daly's paper</a>: the formula and its condition.</li>\n<li><a href=\"https://kubernetes.io/docs/concepts/scheduling-eviction/gang-scheduling/\" target=\"_blank\" rel=\"noopener noreferrer\">Gang scheduling (Kubernetes documentation)</a>: the all-or-nothing rule.</li>\n<li><a href=\"https://trainer.kubeflow.org/en/latest/operator-guides/job-scheduling/coscheduling.html\" target=\"_blank\" rel=\"noopener noreferrer\">Coscheduling (Kubeflow Trainer documentation)</a>: how training pods start together.</li>\n<li><a href=\"https://slurm.schedmd.com/overview.html\" target=\"_blank\" rel=\"noopener noreferrer\">Slurm overview</a>: allocation, and Slurm's other meaning of \"gang scheduling\".</li>\n<li><a href=\"https://github.com/NVIDIA/Megatron-LM/blob/main/tools/preprocess_data.py\" target=\"_blank\" rel=\"noopener noreferrer\">Megatron-LM preprocess_data.py</a>: tokenized binary shards.</li>\n<li><a href=\"https://huggingface.co/docs/safetensors/index\" target=\"_blank\" rel=\"noopener noreferrer\">Safetensors documentation</a>: a tensor format for safe weight storage.</li>\n</ul>","headings":[{"id":"check-yourself","title":"Check yourself","depth":3},{"id":"further-reading","title":"Further reading","depth":3}]}]}