{"id":"design-llm-gateway","title":"Design an LLM gateway (one API in front of many model providers for a whole company)","sections":[{"id":"overview","title":"Overview","html":"<p>A company with 5,000 engineers has about 200 internal apps that call language models: a support bot, a code review assistant, a contract summarizer, nightly classification jobs and many small tools. Every app calls one internal service, the <strong>LLM gateway</strong>, which holds the provider keys, enforces each team's budget, picks a provider for each request and records what it cost. LiteLLM, Portkey and Cloudflare AI Gateway are products of this kind.</p>\n<p>The <a href=\"/learn/llm-gateways\">LLM gateways</a> chapter explains the parts. This chapter applies them to one company with real volumes and makes the decisions an interviewer will probe. The <a href=\"/learn/design-chat-assistant\">chat assistant</a> is a typical caller of this gateway, and the <a href=\"/learn/design-inference-platform\">inference platform</a> is a typical self-hosted target behind it.</p>","headings":[]},{"id":"requirements-and-estimates","title":"Requirements and estimates","html":"<h3 id=\"requirements-and-estimates\">Requirements and estimates</h3>\n<p>Functional requirements:</p>\n<ul>\n<li><strong>One API</strong>, OpenAI-compatible (the request shape most client libraries already speak), with streaming and tool calls.</li>\n<li><strong>Several targets:</strong> hosted providers A and B, and a self-hosted pool run by the inference team.</li>\n<li><strong>Virtual keys:</strong> gateway-issued keys per app, tied to a team and its allowed aliases. Provider keys never leave the gateway.</li>\n<li><strong>Budgets:</strong> a monthly dollar budget and per-minute token limits per team.</li>\n<li><strong>Routing:</strong> aliases, weights, retries, fallbacks and canary rollouts of new model versions.</li>\n<li><strong>Hooks</strong> for redaction and prompt-injection screening, plus audit logs and monthly chargeback.</li>\n</ul>\n<p>Non-functional requirements: the gateway adds at most <strong>20 ms at p99</strong> (the value 99% of requests beat) between receiving a request and sending it upstream. It is 99.95% available, since no app can bypass it. No team overspends by more than a few dollars, and every token is charged to exactly one team.</p>\n<p><strong>Traffic.</strong> Assume 21.6 million requests a day, or 21,600,000 ÷ 86,400 s = <strong>250 a second</strong> on average. Working hours and nightly batch windows give a peak factor of 4, so <strong>1,000 a second</strong> at peak. An average request has a 2,500-token prompt and a 300-token reply, so the peak is 2.5 million input and 300,000 output tokens a second. Per day that is 54 billion input and 6.48 billion output tokens.</p>\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n<div class=\"table-wrap\"><table><thead><tr><th>Alias</th><th>Share</th><th>Requests a day</th><th>Peak a second</th><th>Served by</th></tr></thead><tbody><tr><td><code>chat-large</code></td><td>15%</td><td>3.24 million</td><td>150</td><td>Provider A (70%), Provider B (30%)</td></tr><tr><td><code>chat-small</code></td><td>55%</td><td>11.88 million</td><td>550</td><td>Providers A and B</td></tr><tr><td><code>open-70b</code></td><td>30%</td><td>6.48 million</td><td>300</td><td>Self-hosted pool</td></tr></tbody></table></div>\n<p><strong>Spend.</strong> Assume <code>chat-large</code> costs $3 per million input tokens and $15 per million output tokens, and <code>chat-small</code> $0.25 and $1.25. Both prices are assumptions. Assume half of all input tokens are provider prompt-cache reads at 0.1 times the input price, so <code>chat-large</code> input averages 0.5 × $3 + 0.5 × $0.30 = $1.65 per million.</p>\n<ul>\n<li><code>chat-large</code> per day: 8.1 billion input × $1.65 + 0.972 billion output × $15 = $13,365 + $14,580 = $27,945, or <strong>$838,350</strong> over 30 days.</li>\n<li><code>chat-small</code> per day: 29.7 billion × $0.1375 + 3.564 billion × $1.25 ≈ $4,084 + $4,455 = $8,539, or about <strong>$256,160</strong> a month.</li>\n<li><code>open-70b</code>: with half of each 2,500-token prompt cached, the <a href=\"/learn/design-inference-platform#how-many-requests-one-replica-takes\">inference platform's</a> coupled model plans a 70B replica on four H100s at about 5.2 requests a second, so 300 a second needs 58 replicas plus a spare. The inference team bills 236 H100s × $3 × 730 hours = <strong>$516,840</strong> a month.</li>\n</ul>\n<p>That is about <strong>$1.61 million a month</strong>, or about $2,200 an hour on average. An average <code>chat-large</code> request costs about $0.0086. The worst case, with no cache hits and a reply that runs to the gateway's default cap of 4,096 tokens, is 2,500 × $3 ÷ 10⁶ + 4,096 × $15 ÷ 10⁶ ≈ <strong>$0.069</strong>.</p>\n<p><strong>Instances.</strong> A request lasts about 6 seconds, mostly waiting for tokens, so by <a href=\"/learn/performance-and-latency#littles-law\">Little's law</a> 1,000 × 6 = <strong>6,000 streams</strong> are open at peak. Assume 1.5 ms of CPU per request for parsing, translation and logging, 10 µs for each of about 300 stream events (3 ms), and 2 ms of redaction on the 30% of traffic that asks for it (0.6 ms on average). That is 5.1 ms, or 5.1 cores at peak for a compiled proxy (a Python proxy can need several times more). At 40% CPU, 5.1 ÷ 0.4 ≈ 12.75 cores, or four 4-vCPU instances. Any two of three availability zones must hold four, so each zone needs two, plus one for rolling deploys: <strong>9 instances</strong>.</p>\n<p><strong>Logs.</strong> A 1 KB metadata row per request is <strong>21.6 GB a day</strong>. Bodies average 2,500 × 4 + 300 × 4 = 11,200 bytes, so logging every body would add about 242 GB a day, or 7.3 TB over 30 days, and the 10% of traffic whose owners opt in adds 24 GB a day.</p>","headings":[{"id":"requirements-and-estimates","title":"Requirements and estimates","depth":3}]},{"id":"api-design","title":"API design","html":"<h3 id=\"api-design\">API design</h3>\n<p>Apps call the standard Chat Completions path with a gateway key, so any OpenAI-compatible client works by changing its base URL.</p>\n<pre class=\"shiki shiki-themes github-light github-dark\" style=\"background-color:#fff;--shiki-dark-bg:#24292e;color:#24292e;--shiki-dark:#e1e4e8\" tabindex=\"0\"><code><span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">POST</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> /v1/chat/completions </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">HTTP</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">/</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">1.1</span></span>\n<span class=\"line\"><span style=\"color:#22863A;--shiki-dark:#85E89D\">Authorization</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">:</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> Bearer vk_search-prod_4f1c...</span></span>\n<span class=\"line\"><span style=\"color:#22863A;--shiki-dark:#85E89D\">X-GW-Priority</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">:</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> interactive</span></span>\n<span class=\"line\"><span style=\"color:#22863A;--shiki-dark:#85E89D\">X-GW-Data-Class</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">:</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> confidential</span></span>\n<span class=\"line\"><span style=\"color:#22863A;--shiki-dark:#85E89D\">Content-Type</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">:</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> application/json</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">{</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">\"model\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"chat-large\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">\"stream\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">true</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">\"max_tokens\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">1024</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">,</span></span>\n<span class=\"line\"><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> \"messages\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: [{</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">\"role\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"user\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">\"content\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"Which plan includes SSO?\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">}]}</span></span></code></pre>\n<p><code>model</code> is an <strong>alias</strong>, a name the gateway maps to a list of targets, and apps may pin a version such as <code>chat-large@2026-09</code>. Batch priority is shed first. The data class (<code>public</code>, <code>internal</code>, <code>confidential</code> or <code>restricted</code>) decides which targets are allowed and whether redaction runs. The response names the target in <code>X-GW-Target</code>, and the final stream chunk adds the cost in dollars.</p>\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n<div class=\"table-wrap\"><table><thead><tr><th>Status and type</th><th>Meaning</th><th>Retry?</th></tr></thead><tbody><tr><td>400 <code>unsupported_for_alias</code></td><td>No target supports a requested feature</td><td>No</td></tr><tr><td>429 <code>rate_limited</code> with <code>Retry-After</code></td><td>The team's per-minute limit</td><td>After the wait</td></tr><tr><td>429 <code>budget_exhausted</code>, no <code>Retry-After</code></td><td>The monthly budget is spent</td><td>No</td></tr><tr><td>503 <code>no_capacity</code> with <code>Retry-After</code></td><td>Every target is failing or cooling down</td><td>Once</td></tr></tbody></table></div>\n<p>The two 429s copy the providers' convention: a spend-cap 429 has no <code>Retry-After</code> (see <a href=\"/learn/llm-gateways#routing-and-fallbacks\">routing and fallbacks</a>), so a well-behaved client stops by itself. Keys, budgets and routes change through an admin API, and each change goes to the audit log with its author.</p>","headings":[{"id":"api-design","title":"API design","depth":3}]},{"id":"high-level-design","title":"High-level design","html":"<h3 id=\"high-level-design\">High-level design</h3>\n<figure class=\"diagram\" data-text=\"200 internal apps -> Internal L4 load balancer (HTTPS, SSE); Internal L4 load balancer -> Gateway 9 instances in 3 zones; Gateway 9 instances in 3 zones <-> Guard service screening; Gateway 9 instances in 3 zones -> Provider A hosted; Gateway 9 instances in 3 zones -> Provider B hosted; Gateway 9 instances in 3 zones -> Self-hosted 70B pool; Gateway 9 instances in 3 zones <-> Redis counters, cache (sync every 500 ms); Control plane keys, routes, budgets -> Gateway 9 instances in 3 zones; Gateway 9 instances in 3 zones -> Usage and audit log (a row per request)\"><div class=\"diagram-scroll\"><svg class=\"dg\" viewBox=\"0 0 800 500\" width=\"800\" role=\"img\" aria-label=\"Every app calls the gateway, and only the gateway holds provider keys\"><defs><marker id=\"dg-arrow\" viewBox=\"0 0 10 10\" refX=\"9\" refY=\"5\" markerWidth=\"7\" markerHeight=\"7\" orient=\"auto-start-reverse\"><path class=\"dg-arrowhead\" d=\"M0,1 L9,5 L0,9 z\"></path></marker></defs><line class=\"dg-edge\" x1=\"151\" y1=\"162\" x2=\"167\" y2=\"162\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge\" x1=\"311\" y1=\"162\" x2=\"327\" y2=\"162\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge\" x1=\"400\" y1=\"123\" x2=\"400\" y2=\"73\" marker-end=\"url(#dg-arrow)\" marker-start=\"url(#dg-arrow)\"></line><line class=\"dg-edge\" x1=\"471\" y1=\"135.4\" x2=\"647\" y2=\"69.4\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge\" x1=\"471\" y1=\"162\" x2=\"647\" y2=\"162\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge\" x1=\"471\" y1=\"188.6\" x2=\"647\" y2=\"254.6\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge\" x1=\"330.7\" y1=\"201\" x2=\"144\" y2=\"306\" marker-end=\"url(#dg-arrow)\" marker-start=\"url(#dg-arrow)\"></line><line class=\"dg-edge dg-dashed\" x1=\"400\" y1=\"305\" x2=\"400\" y2=\"201\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge dg-dashed\" x1=\"439.5\" y1=\"199\" x2=\"686.9\" y2=\"431\" marker-end=\"url(#dg-arrow)\"></line><rect class=\"dg-node dg-client\" x=\"12\" y=\"136\" width=\"136\" height=\"52\" rx=\"26\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"80\" y=\"158.5\">200 internal</tspan><tspan x=\"80\" y=\"174.5\">apps</tspan></text><rect class=\"dg-node dg-box\" x=\"172\" y=\"136\" width=\"136\" height=\"52\" rx=\"8\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"240\" y=\"158.5\">Internal L4</tspan><tspan x=\"240\" y=\"174.5\">load balancer</tspan></text><rect class=\"dg-node dg-box\" x=\"332\" y=\"128\" width=\"136\" height=\"68\" rx=\"8\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"400\" y=\"150.5\">Gateway</tspan><tspan x=\"400\" y=\"166.5\">9 instances</tspan><tspan x=\"400\" y=\"182.5\">in 3 zones</tspan></text><rect class=\"dg-node dg-box\" x=\"332\" y=\"16\" width=\"136\" height=\"52\" rx=\"8\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"400\" y=\"38.5\">Guard service</tspan><tspan x=\"400\" y=\"54.5\">screening</tspan></text><rect class=\"dg-node dg-client\" x=\"652\" y=\"16\" width=\"136\" height=\"52\" rx=\"26\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"720\" y=\"38.5\">Provider A</tspan><tspan x=\"720\" y=\"54.5\">hosted</tspan></text><rect class=\"dg-node dg-client\" x=\"652\" y=\"136\" width=\"136\" height=\"52\" rx=\"26\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"720\" y=\"158.5\">Provider B</tspan><tspan x=\"720\" y=\"174.5\">hosted</tspan></text><path class=\"dg-node dg-store\" d=\"M652,258 a68,7 0 0,1 136,0 v48 a68,7 0 0,1 -136,0 z\"></path><path class=\"dg-line\" d=\"M652,258 a68,7 0 0,0 136,0\"></path><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"720\" y=\"281.5\">Self-hosted</tspan><tspan x=\"720\" y=\"297.5\">70B pool</tspan></text><path class=\"dg-node dg-store\" d=\"M12,318 a68,7 0 0,1 136,0 v48 a68,7 0 0,1 -136,0 z\"></path><path class=\"dg-line\" d=\"M12,318 a68,7 0 0,0 136,0\"></path><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"80\" y=\"341.5\">Redis</tspan><tspan x=\"80\" y=\"357.5\">counters, cache</tspan></text><rect class=\"dg-node dg-box\" x=\"332\" y=\"308\" width=\"136\" height=\"68\" rx=\"8\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"400\" y=\"330.5\">Control plane</tspan><tspan x=\"400\" y=\"346.5\">keys, routes,</tspan><tspan x=\"400\" y=\"362.5\">budgets</tspan></text><rect class=\"dg-node dg-queue\" x=\"652\" y=\"436\" width=\"136\" height=\"52\" rx=\"6\"></rect><line class=\"dg-line\" x1=\"772\" y1=\"444\" x2=\"772\" y2=\"480\"></line><line class=\"dg-line\" x1=\"762\" y1=\"444\" x2=\"762\" y2=\"480\"></line><line class=\"dg-line\" x1=\"752\" y1=\"444\" x2=\"752\" y2=\"480\"></line><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"706\" y=\"458.5\">Usage and</tspan><tspan x=\"706\" y=\"474.5\">audit log</tspan></text><rect class=\"dg-edge-bg\" x=\"123\" y=\"114\" width=\"72\" height=\"18\" rx=\"4\"></rect><text class=\"dg-edge-label\" x=\"159\" y=\"127\" text-anchor=\"middle\">HTTPS, SSE</text><rect class=\"dg-edge-bg\" x=\"179.6\" y=\"244.5\" width=\"115.4\" height=\"18\" rx=\"4\"></rect><text class=\"dg-edge-label\" x=\"237.3\" y=\"257.5\" text-anchor=\"middle\">sync every 500 ms</text><rect class=\"dg-edge-bg\" x=\"505.5\" y=\"306\" width=\"115.4\" height=\"18\" rx=\"4\"></rect><text class=\"dg-edge-label\" x=\"563.2\" y=\"319\" text-anchor=\"middle\">a row per request</text></svg></div><figcaption>Every app calls the gateway, and only the gateway holds provider keys</figcaption></figure>\n<ol>\n<li>A layer-4 load balancer (one that forwards TCP connections without reading HTTP) spreads connections by least connections, since streams last seconds.</li>\n<li>The instance looks up the hashed virtual key locally, checks alias, data class and region rules, places a budget hold and checks the exact-match cache if the app opted in.</li>\n<li>It picks a target, translates the request and sends it on a pooled TLS connection, with screening running alongside.</li>\n<li>It relays the stream, settles the hold with the provider's token counts and writes one usage row.</li>\n</ol>\n<p>Instances serve from a versioned snapshot of keys, routes and prices that the control plane pushes, so a control-plane outage only freezes configuration.</p>","headings":[{"id":"high-level-design","title":"High-level design","depth":3}]},{"id":"the-request-path-and-where-time-goes","title":"The request path and where time goes","html":"<h3 id=\"the-request-path-and-where-time-goes\">The request path and where time goes</h3>\n<p>Provider time to first token (TTFT, the wait before the first token) is usually hundreds of milliseconds, but every app pays the gateway's share on every call, so each step gets a budget. The figures are assumptions to replace with measurements.</p>\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n<div class=\"table-wrap\"><table><thead><tr><th>Step</th><th>p99 budget</th></tr></thead><tbody><tr><td>App to gateway, same region, reused HTTP/2 connection</td><td>1.0 ms</td></tr><tr><td>Key lookup, alias, data class and region checks</td><td>0.2 ms</td></tr><tr><td>Parse and validate the JSON body</td><td>1.0 ms</td></tr><tr><td>Budget hold and limits on local counters</td><td>0.2 ms</td></tr><tr><td>Exact-cache lookup in Redis (opted-in apps)</td><td>1.5 ms</td></tr><tr><td>Regex redaction (opted-in apps)</td><td>2.0 ms</td></tr><tr><td>Translate to the provider's format</td><td>1.0 ms</td></tr><tr><td>Queue the usage row (asynchronous)</td><td>0.5 ms</td></tr><tr><td><strong>Total</strong></td><td><strong>7.4 ms</strong></td></tr></tbody></table></div>\n<p>Summing p99s overstates the combined p99, and the headroom covers pauses and very large prompts. Provider connections stay pooled, since a fresh TLS handshake to a distant provider costs tens of milliseconds.</p>\n<figure class=\"diagram\" data-text=\"App request -> Key, alias, data class; Key, alias, data class -> Budget hold, limits; Budget hold, limits -> Route and translate; Route and translate -> Provider; Route and translate -> Injection screen (prompt copy); Provider -> Relay holds first token (tokens); Injection screen -> Relay holds first token (verdict); Relay holds first token -> App sees first token; Relay holds first token -> Settle and log usage (at stream end)\"><div class=\"diagram-scroll\"><svg class=\"dg\" viewBox=\"0 0 800 445\" width=\"800\" role=\"img\" aria-label=\"Screening runs beside the provider call and gates the first token\"><defs><marker id=\"dg-arrow\" viewBox=\"0 0 10 10\" refX=\"9\" refY=\"5\" markerWidth=\"7\" markerHeight=\"7\" orient=\"auto-start-reverse\"><path class=\"dg-arrowhead\" d=\"M0,1 L9,5 L0,9 z\"></path></marker></defs><line class=\"dg-edge\" x1=\"151\" y1=\"42\" x2=\"167\" y2=\"42\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge\" x1=\"311\" y1=\"42\" x2=\"327\" y2=\"42\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge\" x1=\"471\" y1=\"42\" x2=\"487\" y2=\"42\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge\" x1=\"631\" y1=\"42\" x2=\"647\" y2=\"42\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge dg-dashed\" x1=\"534.2\" y1=\"71\" x2=\"427.6\" y2=\"191\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge\" x1=\"720\" y1=\"69\" x2=\"720\" y2=\"191\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge\" x1=\"471\" y1=\"222\" x2=\"647\" y2=\"222\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge\" x1=\"720\" y1=\"251\" x2=\"720\" y2=\"371\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge dg-dashed\" x1=\"668.4\" y1=\"251\" x2=\"464\" y2=\"366\" marker-end=\"url(#dg-arrow)\"></line><rect class=\"dg-node dg-client\" x=\"12\" y=\"18\" width=\"136\" height=\"48\" rx=\"24\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"80\" y=\"46.5\">App request</tspan></text><rect class=\"dg-node dg-box\" x=\"172\" y=\"16\" width=\"136\" height=\"52\" rx=\"8\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"240\" y=\"38.5\">Key, alias,</tspan><tspan x=\"240\" y=\"54.5\">data class</tspan></text><rect class=\"dg-node dg-box\" x=\"332\" y=\"16\" width=\"136\" height=\"52\" rx=\"8\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"400\" y=\"38.5\">Budget hold,</tspan><tspan x=\"400\" y=\"54.5\">limits</tspan></text><rect class=\"dg-node dg-box\" x=\"492\" y=\"16\" width=\"136\" height=\"52\" rx=\"8\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"560\" y=\"38.5\">Route and</tspan><tspan x=\"560\" y=\"54.5\">translate</tspan></text><rect class=\"dg-node dg-client\" x=\"652\" y=\"18\" width=\"136\" height=\"48\" rx=\"24\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"720\" y=\"46.5\">Provider</tspan></text><rect class=\"dg-node dg-box\" x=\"332\" y=\"196\" width=\"136\" height=\"52\" rx=\"8\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"400\" y=\"218.5\">Injection</tspan><tspan x=\"400\" y=\"234.5\">screen</tspan></text><rect class=\"dg-node dg-box\" x=\"652\" y=\"196\" width=\"136\" height=\"52\" rx=\"8\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"720\" y=\"218.5\">Relay holds</tspan><tspan x=\"720\" y=\"234.5\">first token</tspan></text><rect class=\"dg-node dg-client\" x=\"652\" y=\"376\" width=\"136\" height=\"52\" rx=\"26\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"720\" y=\"398.5\">App sees</tspan><tspan x=\"720\" y=\"414.5\">first token</tspan></text><path class=\"dg-node dg-store\" d=\"M332,378 a68,7 0 0,1 136,0 v48 a68,7 0 0,1 -136,0 z\"></path><path class=\"dg-line\" d=\"M332,378 a68,7 0 0,0 136,0\"></path><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"400\" y=\"401.5\">Settle and</tspan><tspan x=\"400\" y=\"417.5\">log usage</tspan></text><rect class=\"dg-edge-bg\" x=\"441.8\" y=\"122\" width=\"78.2\" height=\"18\" rx=\"4\"></rect><text class=\"dg-edge-label\" x=\"480.9\" y=\"135\" text-anchor=\"middle\">prompt copy</text><rect class=\"dg-edge-bg\" x=\"696.4\" y=\"121\" width=\"47.2\" height=\"18\" rx=\"4\"></rect><text class=\"dg-edge-label\" x=\"720\" y=\"134\" text-anchor=\"middle\">tokens</text><rect class=\"dg-edge-bg\" x=\"532.3\" y=\"213\" width=\"53.4\" height=\"18\" rx=\"4\"></rect><text class=\"dg-edge-label\" x=\"559\" y=\"226\" text-anchor=\"middle\">verdict</text><rect class=\"dg-edge-bg\" x=\"520.9\" y=\"299.5\" width=\"90.6\" height=\"18\" rx=\"4\"></rect><text class=\"dg-edge-label\" x=\"566.2\" y=\"312.5\" text-anchor=\"middle\">at stream end</text></svg></div><figcaption>Screening runs beside the provider call and gates the first token</figcaption></figure>\n<p>An injection classifier does not fit the table: assume it takes 25 ms at p99. The gateway starts it alongside the provider call, and the relay holds back the first token until the verdict. TTFT almost always exceeds 25 ms, so users see no extra wait. A flagged request is cancelled before any output reaches the app, costing only its input tokens at an assumed flag rate of 0.3%. A classifier is one layer among several (<a href=\"/learn/ai-security#prompt-injection\">prompt injection</a>).</p>","headings":[{"id":"the-request-path-and-where-time-goes","title":"The request path and where time goes","depth":3}]},{"id":"budgets-across-many-instances","title":"Budgets across many instances","html":"<h3 id=\"budgets-across-many-instances\">Budgets across many instances</h3>\n<p>Take the search team: $20,000 a month on <code>chat-large</code>, with its key capped at 50 requests a second across nine instances. A Redis round trip per request would make Redis a dependency of every call, so each instance counts locally.</p>\n<figure class=\"diagram\" data-text=\"Gateway zone a <-> Redis team totals (+pending, total); Gateway zone b <-> Redis team totals; Gateway zone c <-> Redis team totals; Ledger settled cost -> Redis team totals (nightly true-up)\"><div class=\"diagram-scroll\"><svg class=\"dg\" viewBox=\"0 0 800 320\" width=\"800\" role=\"img\" aria-label=\"Instances count locally and push changes every 500 ms\"><defs><marker id=\"dg-arrow\" viewBox=\"0 0 10 10\" refX=\"9\" refY=\"5\" markerWidth=\"7\" markerHeight=\"7\" orient=\"auto-start-reverse\"><path class=\"dg-arrowhead\" d=\"M0,1 L9,5 L0,9 z\"></path></marker></defs><line class=\"dg-edge\" x1=\"153\" y1=\"69.4\" x2=\"327\" y2=\"134.6\" marker-end=\"url(#dg-arrow)\" marker-start=\"url(#dg-arrow)\"></line><line class=\"dg-edge\" x1=\"153\" y1=\"162\" x2=\"327\" y2=\"162\" marker-end=\"url(#dg-arrow)\" marker-start=\"url(#dg-arrow)\"></line><line class=\"dg-edge\" x1=\"153\" y1=\"254.6\" x2=\"327\" y2=\"189.4\" marker-end=\"url(#dg-arrow)\" marker-start=\"url(#dg-arrow)\"></line><line class=\"dg-edge dg-dashed\" x1=\"649\" y1=\"162\" x2=\"473\" y2=\"162\" marker-end=\"url(#dg-arrow)\"></line><rect class=\"dg-node dg-box\" x=\"12\" y=\"16\" width=\"136\" height=\"52\" rx=\"8\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"80\" y=\"38.5\">Gateway</tspan><tspan x=\"80\" y=\"54.5\">zone a</tspan></text><rect class=\"dg-node dg-box\" x=\"12\" y=\"136\" width=\"136\" height=\"52\" rx=\"8\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"80\" y=\"158.5\">Gateway</tspan><tspan x=\"80\" y=\"174.5\">zone b</tspan></text><rect class=\"dg-node dg-box\" x=\"12\" y=\"256\" width=\"136\" height=\"52\" rx=\"8\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"80\" y=\"278.5\">Gateway</tspan><tspan x=\"80\" y=\"294.5\">zone c</tspan></text><path class=\"dg-node dg-store\" d=\"M332,138 a68,7 0 0,1 136,0 v48 a68,7 0 0,1 -136,0 z\"></path><path class=\"dg-line\" d=\"M332,138 a68,7 0 0,0 136,0\"></path><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"400\" y=\"161.5\">Redis</tspan><tspan x=\"400\" y=\"177.5\">team totals</tspan></text><path class=\"dg-node dg-store\" d=\"M652,138 a68,7 0 0,1 136,0 v48 a68,7 0 0,1 -136,0 z\"></path><path class=\"dg-line\" d=\"M652,138 a68,7 0 0,0 136,0\"></path><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"720\" y=\"161.5\">Ledger</tspan><tspan x=\"720\" y=\"177.5\">settled cost</tspan></text><rect class=\"dg-edge-bg\" x=\"188.5\" y=\"93\" width=\"103\" height=\"18\" rx=\"4\"></rect><text class=\"dg-edge-label\" x=\"240\" y=\"106\" text-anchor=\"middle\">+pending, total</text><rect class=\"dg-edge-bg\" x=\"509.5\" y=\"153\" width=\"103\" height=\"18\" rx=\"4\"></rect><text class=\"dg-edge-label\" x=\"561\" y=\"166\" text-anchor=\"middle\">nightly true-up</text></svg></div><figcaption>Instances count locally and push changes every 500 ms</figcaption></figure>\n<p>Each instance keeps a <strong>pending</strong> amount: holds it has placed plus settlement adjustments since its last sync. Every 500 ms it adds pending to the shared total in Redis and reads back the new total in the same call. A hold is the worst case, and settlement returns the unused part, as in <a href=\"/learn/llm-gateways#token-based-rate-limits\">token-based rate limits</a>.</p>\n<pre class=\"shiki shiki-themes github-light github-dark\" style=\"background-color:#fff;--shiki-dark-bg:#24292e;color:#24292e;--shiki-dark:#e1e4e8\" tabindex=\"0\"><code><span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">class</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\"> TeamBudget</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">:</span></span>\n<span class=\"line\"><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">    \"\"\"One instance's view of a team's monthly budget, in dollars.\"\"\"</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    def</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> __init__</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(self, store: </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">dict</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, team: </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">str</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, limit: </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">float</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">):</span></span>\n<span class=\"line\"><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">        self</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">.store, </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">self</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">.team, </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">self</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">.limit </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> store, team, limit</span></span>\n<span class=\"line\"><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">        self</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">.seen </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> store.get(team, </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">0.0</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">)   </span><span style=\"color:#6A737D;--shiki-dark:#6A737D\"># shared total at the last sync</span></span>\n<span class=\"line\"><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">        self</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">.pending </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 0.0</span><span style=\"color:#6A737D;--shiki-dark:#6A737D\">                 # holds and settlements not yet pushed</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    def</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\"> reserve</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(self, worst: </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">float</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">) -> </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">bool</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">:</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">        if</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> self</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">.seen </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">+</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> self</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">.pending </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">+</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> worst </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">></span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> self</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">.limit:</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">            return</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> False</span><span style=\"color:#6A737D;--shiki-dark:#6A737D\">                   # 429 budget_exhausted</span></span>\n<span class=\"line\"><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">        self</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">.pending </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">+=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> worst</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">        return</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> True</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    def</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\"> settle</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(self, worst: </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">float</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, actual: </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">float</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">) -> </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">None</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">:</span></span>\n<span class=\"line\"><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">        self</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">.pending </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">+=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> actual </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">-</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> worst     </span><span style=\"color:#6A737D;--shiki-dark:#6A737D\"># give back the unused part</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    def</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\"> sync</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(self) -> </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">None</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">:                </span><span style=\"color:#6A737D;--shiki-dark:#6A737D\"># Redis: one INCRBYFLOAT returns the total</span></span>\n<span class=\"line\"><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">        self</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">.store[</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">self</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">.team] </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> self</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">.store.get(</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">self</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">.team, </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">0.0</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">) </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">+</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> self</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">.pending</span></span>\n<span class=\"line\"><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">        self</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">.seen, </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">self</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">.pending </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> self</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">.store[</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">self</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">.team], </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">0.0</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">store </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> {</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"search\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">19_999.90</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">}              </span><span style=\"color:#6A737D;--shiki-dark:#6A737D\"># 10 cents left of $20,000</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">a, b </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> TeamBudget(store, </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"search\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">20_000</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">), TeamBudget(store, </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"search\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">20_000</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">)</span></span>\n<span class=\"line\"><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">print</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(a.reserve(</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">0.069</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">), b.reserve(</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">0.069</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">))  </span><span style=\"color:#6A737D;--shiki-dark:#6A737D\"># True True: neither sees the other</span></span>\n<span class=\"line\"><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">print</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(a.reserve(</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">0.069</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">))                    </span><span style=\"color:#6A737D;--shiki-dark:#6A737D\"># False: a's own hold counts</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">a.settle(</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">0.069</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">0.069</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">); b.settle(</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">0.069</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">0.069</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">)</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">a.sync(); b.sync()</span></span>\n<span class=\"line\"><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">print</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">round</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(store[</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"search\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">] </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">-</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 20_000</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">3</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">))  </span><span style=\"color:#6A737D;--shiki-dark:#6A737D\"># 0.038: the overshoot</span></span></code></pre>\n<p>The overshoot has a ceiling. In one 500 ms interval, all instances together admit at most 50 × 0.5 = 25 requests the others cannot see. At the worst case of $0.069 each that is 25 × $0.069 ≈ <strong>$1.72</strong>, about 0.009% of the budget, and at the average of $0.0086 it is about $0.22.</p>\n<p>Syncing holds along with settled spend keeps that window short. If only settled spend were synced, each request would stay invisible for its whole 5-second stream plus the interval, and the bound would grow to 50 × 5.5 = 275 requests × $0.069 ≈ $19.</p>\n<p>Per-minute token limits use the same counters, reset each minute. When a provider quota is crossed, instances may admit for up to 500 ms more, and the provider answers the excess with 429s that the router handles. The <a href=\"/learn/rate-limiting#distributed-counters-in-redis\">distributed counters</a> section covers the atomic Redis operations. Each night the ledger recomputes month-to-date spend and corrects the Redis totals, so a lost sync drifts for at most a day.</p>","headings":[{"id":"budgets-across-many-instances","title":"Budgets across many instances","depth":3}]},{"id":"routing-retries-and-fallbacks","title":"Routing, retries and fallbacks","html":"<h3 id=\"routing-retries-and-fallbacks\">Routing, retries and fallbacks</h3>\n<p>Each alias lists targets with a weight, features (tools, JSON schema output, context length), a data-class ceiling and a region. The router drops targets that cannot serve the request or are cooling down, then chooses by weight.</p>\n<p>Weighted random choice scatters an app's requests across providers, each with its own prompt cache. Assume an app sends a request every 4 minutes with the same 6,000-token system prompt to providers whose caches last 5 minutes and charge 1.25 times the input price to write. On one provider nearly every request hits, at 6,000 × $0.30 ÷ 10⁶ = $0.0018. Split 50/50, each provider sees the app every 8 minutes, so most requests miss and pay a write: 6,000 × $3.75 ÷ 10⁶ = $0.0225, 12.5 times as much.</p>\n<p>The router therefore uses <strong>weighted rendezvous hashing</strong>: each target scores a hash of the app's key and the target's name, scaled by its weight (score = −weight ÷ ln(hash), with the hash mapped into (0, 1)), and the highest score wins. The same app always lands on the same target, traffic still splits by weight across apps (6,947 of 10,000 test keys chose a target weighted 70 against 30), and when a target cools down only its own apps move. The sticky key is the app plus a hash of its system prompt, and a hot key spills past a capped share to its next-best target.</p>\n<p><strong>Health.</strong> A 429 with <code>Retry-After: 60</code> cools that target (provider, model and key) down for 60 seconds, shared through Redis so all instances stop within one sync. A <a href=\"/learn/reliability-patterns#circuit-breakers\">circuit breaker</a> opens when 50% of at least 20 requests in 10 seconds fail, and lets probes through after 30 seconds. A spend-cap 429 takes the target out until a person clears it.</p>\n<p><strong>Retry storms.</strong> Provider A carries 70% of <code>chat-large</code> at peak, about 105 requests a second. If it fails and both the app's SDK and the gateway make three attempts, A receives up to 105 × 3 × 3 = 945 requests a second. Apps therefore disable SDK retries, leaving one retry layer (<a href=\"/learn/reliability-patterns#retry-storms-and-retry-budgets\">retry storms and retry budgets</a>), and each target gets a <strong>retry budget</strong>:</p>\n<pre class=\"shiki shiki-themes github-light github-dark\" style=\"background-color:#fff;--shiki-dark-bg:#24292e;color:#24292e;--shiki-dark:#e1e4e8\" tabindex=\"0\"><code><span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">class</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\"> RetryBudget</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">:</span></span>\n<span class=\"line\"><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">    \"\"\"Allow retries worth `ratio` of first attempts, plus a small burst.\"\"\"</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    def</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> __init__</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(self, ratio</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">0.1</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, burst</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">10</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">):</span></span>\n<span class=\"line\"><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">        self</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">.ratio, </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">self</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">.burst, </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">self</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">.tokens </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> ratio, burst, burst</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    def</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\"> first_attempt</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(self):</span></span>\n<span class=\"line\"><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">        self</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">.tokens </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> min</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">self</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">.burst, </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">self</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">.tokens </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">+</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> self</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">.ratio)</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    def</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\"> may_retry</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(self):</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">        if</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> self</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">.tokens </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">>=</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 1</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">:</span></span>\n<span class=\"line\"><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">            self</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">.tokens </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">-=</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 1</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">            return</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> True</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">        return</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> False</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">budget, calls </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> RetryBudget(), </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">0</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">for</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> _ </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">in</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> range</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">1_000</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">):                     </span><span style=\"color:#6A737D;--shiki-dark:#6A737D\"># the target fails every call</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">    budget.first_attempt()</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">    calls </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">+=</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 1</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    for</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> _ </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">in</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> range</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">2</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">):                     </span><span style=\"color:#6A737D;--shiki-dark:#6A737D\"># up to two retries each</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">        if</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> not</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> budget.may_retry():</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">            break</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">        calls </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">+=</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 1</span></span>\n<span class=\"line\"><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">print</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(calls)                               </span><span style=\"color:#6A737D;--shiki-dark:#6A737D\"># 1109, against 3,000 without a budget</span></span></code></pre>\n<p>A request that may not retry goes straight to its fallback.</p>\n<p><strong>Fallback capacity.</strong> At peak, <code>chat-large</code> sends 150 × 2,500 × 60 = 22.5 million input tokens a minute. If the company's quota at Provider B is 8 million a minute (assumed), B can absorb 8,000,000 ÷ (2,500 × 60) ≈ 53 requests a second, about 36% of the alias. If a third of <code>chat-large</code> is interactive (50 a second), B can take all of it during an outage at A, while batch waits. Ramp the shift over a minute or two, since Anthropic warns that sharp usage increases can trigger acceleration limits.</p>","headings":[{"id":"routing-retries-and-fallbacks","title":"Routing, retries and fallbacks","depth":3}]},{"id":"streaming-and-failover","title":"Streaming and failover","html":"<h3 id=\"streaming-and-failover\">Streaming and failover</h3>\n<p>A streamed reply's status and headers go out once, before the first event, and the gateway delays them until the provider's first content event. The user sees nothing before the first token either way, so this costs nothing, and every failure before that point stays retryable on another target: 429s, 529 overloads, connection errors and stalled prefills. A <strong>first-token deadline</strong> catches stalls. Prefill grows with prompt length, so for <code>chat-large</code> assume 4 seconds plus 50 ms per 1,000 prompt tokens (about 4.1 seconds for 2,500 tokens, 9 seconds for 100,000), set above each target's measured p99.9.</p>\n<p>After tokens have reached the app, a second provider would start the answer over while the user has read part of the first. The gateway ends the stream with an error event and closes it without the final <code>data: [DONE]</code> line, so clients can tell the reply is incomplete. Resuming from the partial text is possible (see <a href=\"/learn/llm-gateways#streaming-through-a-gateway\">streaming through a gateway</a>), but that choice belongs to apps that can tolerate a seam.</p>\n<p>OpenAI-style streams report usage only in a final chunk, and only when <code>stream_options.include_usage</code> is set, so the gateway always sets it. If a stream breaks before that chunk, the gateway estimates output from the relayed text and marks the row for the daily reconciliation against the provider's usage export.</p>","headings":[{"id":"streaming-and-failover","title":"Streaming and failover","depth":3}]},{"id":"adapters-what-differs-between-providers","title":"Adapters: what differs between providers","html":"<h3 id=\"adapters-what-differs-between-providers\">Adapters: what differs between providers</h3>\n<p>The adapter layer translates requests and responses for each provider. Most of the work sits in a few differences:</p>\n<ul>\n<li><strong>System prompt and limits.</strong> OpenAI puts the system prompt in the message list. Anthropic's Messages API takes a top-level <code>system</code> field and requires <code>max_tokens</code>, so the gateway sets a default of 4,096 per alias, which also bounds the budget hold.</li>\n<li><strong>Tool calls.</strong> OpenAI returns <code>tool_calls</code> whose arguments are a JSON string. Anthropic returns <code>tool_use</code> blocks whose <code>input</code> is an object and expects <code>tool_result</code> blocks back. Both stream arguments as partial JSON, so the adapter assembles fragments per call. It keeps tool call IDs as opaque strings, since a conversation may switch provider between turns.</li>\n<li><strong>Tokenizers.</strong> The same text gives different token counts on different models, so budgets are kept in dollars and per-minute limits in each target's own tokens.</li>\n<li><strong>Prompt caching.</strong> OpenAI caches long prefixes automatically. Anthropic caches at <code>cache_control</code> breakpoints and charges to write, so the adapter marks one after long, stable system prompts. The gateway never inserts per-request text ahead of an app's prompt, since that breaks every prefix match.</li>\n<li><strong>Usage fields.</strong> OpenAI's <code>prompt_tokens</code> includes cached tokens. Anthropic's <code>input_tokens</code> excludes cache reads and writes, which arrive in separate fields.</li>\n</ul>\n<pre class=\"shiki shiki-themes github-light github-dark\" style=\"background-color:#fff;--shiki-dark-bg:#24292e;color:#24292e;--shiki-dark:#e1e4e8\" tabindex=\"0\"><code><span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">def</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\"> normalize</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(provider: </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">str</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, u: </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">dict</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">) -> </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">dict</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">:</span></span>\n<span class=\"line\"><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">    \"\"\"Map a provider's usage block to the ledger's four token kinds.\"\"\"</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    if</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> provider </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">==</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> \"openai\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">:               </span><span style=\"color:#6A737D;--shiki-dark:#6A737D\"># prompt_tokens includes cached tokens</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">        cached </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> u.get(</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"prompt_tokens_details\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, {}).get(</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"cached_tokens\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">0</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">)</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">        return</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> {</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"uncached_in\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: u[</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"prompt_tokens\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">] </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">-</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> cached, </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"cache_read\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: cached,</span></span>\n<span class=\"line\"><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">                \"cache_write\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">0</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"out\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: u[</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"completion_tokens\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">]}</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    if</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> provider </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">==</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> \"anthropic\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">:            </span><span style=\"color:#6A737D;--shiki-dark:#6A737D\"># input_tokens excludes cache reads and writes</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">        return</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> {</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"uncached_in\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: u[</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"input_tokens\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">],</span></span>\n<span class=\"line\"><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">                \"cache_read\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: u.get(</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"cache_read_input_tokens\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">0</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">),</span></span>\n<span class=\"line\"><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">                \"cache_write\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: u.get(</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"cache_creation_input_tokens\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">0</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">),</span></span>\n<span class=\"line\"><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">                \"out\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: u[</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"output_tokens\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">]}</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    raise</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> ValueError</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">f</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"no adapter for </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">{</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">provider</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">}</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">)</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">print</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(normalize(</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"openai\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, {</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"prompt_tokens\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">2500</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"completion_tokens\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">300</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">,</span></span>\n<span class=\"line\"><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">                           \"prompt_tokens_details\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: {</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"cached_tokens\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">1500</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">}}))</span></span>\n<span class=\"line\"><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">print</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(normalize(</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"anthropic\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, {</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"input_tokens\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">1000</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"cache_read_input_tokens\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">1500</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">,</span></span>\n<span class=\"line\"><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">                              \"output_tokens\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">300</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">}))</span></span>\n<span class=\"line\"><span style=\"color:#6A737D;--shiki-dark:#6A737D\"># Both: {'uncached_in': 1000, 'cache_read': 1500, 'cache_write': 0, 'out': 300}</span></span></code></pre>\n<p>Reading Anthropic's <code>input_tokens</code> as the whole prompt would record 1,000 tokens for a 2,500-token prompt. Each adapter has contract tests on recorded responses and a nightly live test. Provider-only features go in a <code>provider_options</code> field, which pins the request to that provider and disables fallback.</p>","headings":[{"id":"adapters-what-differs-between-providers","title":"Adapters: what differs between providers","depth":3}]},{"id":"caching","title":"Caching","html":"<h3 id=\"caching\">Caching</h3>\n<p><strong>Provider prompt caches</strong> are the biggest lever, and the gateway helps by keeping prefixes stable and routing sticky. On <code>chat-large</code>, raising the hit rate from 20% to 50% of input tokens cuts daily input cost from 8,100 × (0.8 × $3 + 0.2 × $0.30) = $19,926 to $13,365, a saving of $6,561 a day, or about <strong>$197,000 a month</strong>.</p>\n<p><strong>An exact-match cache</strong> in the gateway is opt-in per app, with a TTL (time to live). Its key includes the team, the app, the resolved model version and every field that can change the reply (see <a href=\"/learn/llm-gateways#caching-responses\">caching responses</a>), so an old version's answers are never served after a rollout. Assume 8% of requests opt in (eval reruns, repeated classifications) and 40% of those hit, so 3.2% of requests come from Redis. A day of entries is about 1.73 million replies of 1.5 KB, or 2.6 GB.</p>\n<p><strong>A semantic cache</strong>, which matches questions by meaning, can return a confident answer to a different question. It is off by default, allowed only on narrow FAQ routes with a threshold tuned on labeled pairs, scoped to one app and never for personal data, since shared caches leak (<a href=\"/learn/ai-security#logs-traces-and-caches\">logs, traces and caches</a>).</p>","headings":[{"id":"caching","title":"Caching","depth":3}]},{"id":"aliases-and-safe-rollouts","title":"Aliases and safe rollouts","html":"<h3 id=\"aliases-and-safe-rollouts\">Aliases and safe rollouts</h3>\n<p>A new version behind <code>chat-large</code> must pass every registered team's eval set (<a href=\"/learn/ai-evals-and-observability#running-evals-before-release\">running evals before release</a>). It then takes 1% of traffic, sticky by conversation so one conversation never switches models between turns. At an average of 37.5 <code>chat-large</code> requests a second, 1% is 32,400 requests a day. If the error rate rose from 0.5% to 1%, the day would show 324 errors instead of 162, a clear signal. Traffic then moves to 5%, 25% and 100%, a day per step, comparing errors, refusals, output length (which drives cost), TTFT and sampled quality scores. Rollback is a route change.</p>","headings":[{"id":"aliases-and-safe-rollouts","title":"Aliases and safe rollouts","depth":3}]},{"id":"security-residency-and-logging","title":"Security, residency and logging","html":"<h3 id=\"security-residency-and-logging\">Security, residency and logging</h3>\n<p><strong>Keys.</strong> Provider keys live in a secrets manager (see <a href=\"/learn/security#secrets-management\">secrets management</a>) and are never logged, with one provider workspace per business unit to limit what a leak exposes. Virtual keys are stored as SHA-256 hashes with an owner and an expiry, and one config push revokes them.</p>\n<p><strong>Data classes and regions</strong> are hard filters applied before weights. <code>restricted</code> goes only to the self-hosted pool. <code>confidential</code> goes only to targets under a zero-data-retention agreement, in which the provider keeps no copy after answering; check which endpoints and features each agreement covers. EU apps use only EU endpoints, and fallbacks never cross regions, so an outage returns 503 instead of sending data abroad.</p>\n<p><strong>Redaction</strong> protects personal data and can weaken answers: a model drafting a reply to <code>&#x3C;PERSON_1></code> writes a worse email than one that knows the name. The gateway uses reversible placeholders, keeps the mapping in memory for the request and restores names in the reply. A placeholder can arrive split across two stream events, so the relay holds back the last 16 characters until it knows they are not the start of one. Detectors miss some items and flag others, so redaction is opt-in.</p>\n<p><strong>Logging.</strong> Metadata is always logged. Bodies are off by default; when an owner turns them on, they are redacted, encrypted and kept 30 days, and every read is audited, following <a href=\"/learn/ai-evals-and-observability#content-and-privacy\">content and privacy</a>.</p>","headings":[{"id":"security-residency-and-logging","title":"Security, residency and logging","depth":3}]},{"id":"keeping-the-gateway-up","title":"Keeping the gateway up","html":"<h3 id=\"keeping-the-gateway-up\">Keeping the gateway up</h3>\n<p>Every app depends on the gateway, so instances are stateless and span three zones, sized so two zones carry peak. Interactive and batch traffic run in separate deployments, so a batch flood or a bad batch config cannot touch customer-facing apps, the <a href=\"/learn/reliability-patterns#bulkheads\">bulkhead</a> pattern. A route change goes to one instance, then one zone, then all, and rolls back if errors rise. Replies are capped at 4,096 tokens, about 137 seconds at 30 tokens a second, so a deploy drains each instance for 180 seconds.</p>\n<p><strong>Redis outage.</strong> Instances cannot sync budgets. They can <strong>fail closed</strong>, rejecting what they cannot verify, or <strong>fail open</strong>, admitting anyway. Failing closed for half an hour stops 200 apps to prevent at most about $1,100 of extra spend. The gateway fails open with local caps: each instance lets each team spend at most twice its trailing-hour rate divided by the number of instances. Over 30 minutes the company spends at most 2 × $2,200 × 0.5 = $2,200, half of it ordinary traffic. Keys marked as hard caps fail closed. When Redis returns, instances push their local totals before admitting more.</p>","headings":[{"id":"keeping-the-gateway-up","title":"Keeping the gateway up","depth":3}]},{"id":"observability-and-chargeback","title":"Observability and chargeback","html":"<h3 id=\"observability-and-chargeback\">Observability and chargeback</h3>\n<p>The dashboard shows <strong>gateway overhead</strong> as its own histogram, alerting at 15 ms p99 before the 20 ms target is breached. Per target it shows error classes, TTFT, time per output token and breaker state. Per team it shows spend against budget (alerts at 50%, 80% and 100%, plus a burn-rate alert) and cache hit rates. Each request is one trace using the <a href=\"/learn/ai-evals-and-observability#the-genai-attributes\">GenAI attributes</a>.</p>\n<p><strong>Chargeback</strong> comes from the ledger, one row per request with tokens by kind, team, target and <code>price_version</code> (<a href=\"/learn/llm-gateways#tracking-cost\">tracking cost</a>). Hosted requests are charged at provider prices. The self-hosted pool has a fixed cost, so it gets an internal price each quarter. It serves 194.4 million requests a month: 486 billion input and 58.32 billion output tokens. Weighting an output token as five input tokens, like the hosted prices, gives 486 + 5 × 58.32 = 777.6 billion units, so $516,840 works out to about $0.66 per million input tokens and $3.32 per million output tokens.</p>","headings":[{"id":"observability-and-chargeback","title":"Observability and chargeback","depth":3}]},{"id":"trade-offs-to-discuss","title":"Trade-offs to discuss","html":"<h3 id=\"trade-offs-to-discuss\">Trade-offs to discuss</h3>\n<ul>\n<li><strong>Build, run open source or buy.</strong> LiteLLM covers most of this design, with some features behind its enterprise license. A managed gateway removes operations work and sends prompts through another company's network (<a href=\"/learn/llm-gateways#build-or-buy\">build or buy</a>).</li>\n<li><strong>Local counters or a central check.</strong> Local counters keep Redis off the request path, at about $1.72 of overshoot per team per interval. A central check is exact and adds a dependency to every call.</li>\n<li><strong>Sticky or random routing.</strong> Stickiness keeps prompt caches warm, worth about $197,000 a month here. Random routing balances load more evenly.</li>\n<li><strong>Parallel or serial screening.</strong> Parallel hides the classifier's latency and pays for flagged prompts. Serial adds its latency to every call.</li>\n</ul>","headings":[{"id":"trade-offs-to-discuss","title":"Trade-offs to discuss","depth":3}]},{"id":"check-yourself","title":"Check yourself","html":"<h3 id=\"check-yourself\">Check yourself</h3>\n<details>\n<summary>Nine instances enforce a $20,000 budget, sync every 500 ms and hold $0.069 per request, and the team's key is capped at 50 requests a second. How far can the team overshoot? What if instances synced only settled spend?</summary>\n<p>In one interval the instances admit at most 50 × 0.5 = 25 requests the others cannot see, so at most 25 × $0.069 ≈ $1.72. Syncing only settled spend hides each request for its 5-second stream too, so the bound becomes 275 × $0.069 ≈ $19.</p>\n</details>\n<details>\n<summary>Provider A carries 105 requests a second and fails every call. App SDKs and the gateway each make three attempts. How much traffic reaches A, and what is the fix?</summary>\n<p>Up to 105 × 3 × 3 = 945 requests a second, nine times normal. Make the gateway the only retry layer and cap its retries at about 10% of first attempts. Then 1,000 failing requests cause about 1,109 calls instead of 3,000, and the rest go to the fallback.</p>\n</details>\n<details>\n<summary>A stream has sent 40 tokens when Provider A returns an overload error. Provider B is healthy. Should the gateway switch?</summary>\n<p>Not silently, because B would start the answer over. The gateway ends the stream with an error event and no <code>[DONE]</code> line, and the app decides. Failures before the first token are different: the gateway delays headers until the first content event, so those requests move to B unseen.</p>\n</details>\n<details>\n<summary>Redis is down for 30 minutes. What happens to budget checks?</summary>\n<p>Ordinary keys fail open with local caps of twice each team's trailing-hour rate divided by the instance count, so the company spends at most about $2,200 in that half hour, half of it normal traffic. Hard-cap keys fail closed.</p>\n</details>\n<details>\n<summary>The ledger shows Provider B's input tokens about 60% below the team's estimates, only for requests with long system prompts. What is a likely cause?</summary>\n<p>The adapter reads Anthropic-style <code>input_tokens</code> as the whole prompt, but that field excludes cache reads and writes, so a 2,500-token prompt with 1,500 cached tokens records 1,000. Normalize usage into the four token kinds and test on recorded responses.</p>\n</details>\n<h3 id=\"further-reading\">Further reading</h3>\n<ul>\n<li><a href=\"https://docs.litellm.ai/docs/routing\" target=\"_blank\" rel=\"noopener noreferrer\">LiteLLM routing</a> and <a href=\"https://docs.litellm.ai/docs/proxy/users\" target=\"_blank\" rel=\"noopener noreferrer\">virtual keys and budgets</a>: fallbacks, cooldowns, keys and budgets.</li>\n<li><a href=\"https://theagentrouter.ai/\" target=\"_blank\" rel=\"noopener noreferrer\">Agent Router</a>: the Envoy-based gateway formerly called Envoy AI Gateway.</li>\n<li><a href=\"https://platform.claude.com/docs/en/api/rate-limits\" target=\"_blank\" rel=\"noopener noreferrer\">Rate limits</a>, <a href=\"https://platform.claude.com/docs/en/api/errors\" target=\"_blank\" rel=\"noopener noreferrer\">errors</a> and <a href=\"https://platform.claude.com/docs/en/build-with-claude/prompt-caching\" target=\"_blank\" rel=\"noopener noreferrer\">prompt caching</a> (Claude Platform docs): token and acceleration limits, 429 and 529, cache usage fields.</li>\n<li><a href=\"https://developers.openai.com/api/docs/guides/rate-limits\" target=\"_blank\" rel=\"noopener noreferrer\">Rate limits</a> and <a href=\"https://developers.openai.com/api/docs/guides/prompt-caching\" target=\"_blank\" rel=\"noopener noreferrer\">prompt caching</a> (OpenAI developer docs): RPM, TPM and automatic prefix caching.</li>\n<li><a href=\"https://sre.google/sre-book/handling-overload/\" target=\"_blank\" rel=\"noopener noreferrer\">Google SRE book, Handling Overload</a> and <a href=\"https://sre.google/sre-book/addressing-cascading-failures/\" target=\"_blank\" rel=\"noopener noreferrer\">Addressing Cascading Failures</a>: retry budgets and load shedding.</li>\n<li><a href=\"https://aws.amazon.com/builders-library/timeouts-retries-and-backoff-with-jitter/\" target=\"_blank\" rel=\"noopener noreferrer\">AWS Builders' Library, Timeouts, retries, and backoff with jitter</a>: how retries amplify load.</li>\n<li><a href=\"https://github.com/microsoft/presidio\" target=\"_blank\" rel=\"noopener noreferrer\">Microsoft Presidio</a>: open-source detection and anonymization of personal data.</li>\n<li><a href=\"https://genai.owasp.org/llmrisk/llm102025-unbounded-consumption/\" target=\"_blank\" rel=\"noopener noreferrer\">OWASP LLM10: Unbounded Consumption (2025)</a>: runaway spending and its limits.</li>\n</ul>","headings":[{"id":"check-yourself","title":"Check yourself","depth":3},{"id":"further-reading","title":"Further reading","depth":3}]}]}