{"id":"ai-agents","title":"Agents and tool use","sections":[{"id":"overview","title":"Overview","html":"<p>A model call takes a prompt and returns text. It cannot look up an order, send an email or issue a refund, because it has no connection to those systems. An <strong>agent</strong> adds that connection. It is a program that calls a model in a loop. Each time the model replies, the program either accepts the reply as the final answer, or runs the tool the model asked for and returns the result. The model chooses each step. The program decides what each tool does and sets the limits.</p>\n<p>The loop makes an agent useful, and it makes one hard to run. Every step is a model call with its own cost and failure modes, and every tool call may change something outside the program. This chapter covers tool calling, the loop's limits, how context grows, how tools connect through the Model Context Protocol (MCP), and how to run long tasks reliably, safely and affordably. The code is a small loop with a scripted fake model, so it runs without an API key. It assumes you know what a model call costs, which <a href=\"/learn/llm-inference\">How LLM inference works</a> explains.</p>","headings":[]},{"id":"from-a-model-call-to-an-agent","title":"From a model call to an agent","html":"<h3 id=\"from-a-model-call-to-an-agent\">From a model call to an agent</h3>\n<p>Compare two requests. \"Where is my order?\" needs one model call, if the order is already in the prompt. \"Refund order A-1042 if it is eligible\" needs a lookup first, and the next step depends on what the lookup returns. No fixed list of steps covers every case, so the program cannot know in advance how many calls it needs.</p>\n<p>Anthropic's guide to building effective agents, dated December 19, 2024, separates two designs. In a <strong>workflow</strong>, \"LLMs and tools are orchestrated through predefined code paths.\" In an <strong>agent</strong>, LLMs dynamically direct \"their own processes and tool usage.\" The guide warns that \"the autonomous nature of agents means higher costs, and the potential for compounding errors.\"</p>\n<p>In this chapter, an agent is a model, a set of tools and a loop that runs until the task ends or a limit stops it. Start with the simplest design that works: one call when the answer is in the prompt, a fixed workflow when the steps are known, and a loop only when the next step depends on what the model finds.</p>","headings":[{"id":"from-a-model-call-to-an-agent","title":"From a model call to an agent","depth":3}]},{"id":"tool-calling","title":"Tool calling","html":"<h3 id=\"tool-calling\">Tool calling</h3>\n<p>A <strong>tool</strong> is a function that the application runs for the model, together with a description the model reads. Each tool has a name, a description and an input schema written in JSON Schema. Here is an invented refund tool in Anthropic's format:</p>\n<pre class=\"shiki shiki-themes github-light github-dark\" style=\"background-color:#fff;--shiki-dark-bg:#24292e;color:#24292e;--shiki-dark:#e1e4e8\" tabindex=\"0\"><code><span class=\"line\"><span>{</span></span>\n<span class=\"line\"><span>  \"name\": \"issue_refund\",</span></span>\n<span class=\"line\"><span>  \"description\": \"Refund part or all of one order. Call lookup_order first, and refund only orders whose status is delivered.\",</span></span>\n<span class=\"line\"><span>  \"input_schema\": {</span></span>\n<span class=\"line\"><span>    \"type\": \"object\",</span></span>\n<span class=\"line\"><span>    \"properties\": {</span></span>\n<span class=\"line\"><span>      \"order_id\": {\"type\": \"string\"},</span></span>\n<span class=\"line\"><span>      \"amount_cents\": {\"type\": \"integer\", \"minimum\": 1}</span></span>\n<span class=\"line\"><span>    },</span></span>\n<span class=\"line\"><span>    \"required\": [\"order_id\", \"amount_cents\"]</span></span>\n<span class=\"line\"><span>  }</span></span>\n<span class=\"line\"><span>}</span></span></code></pre>\n<p>The description matters as much as the schema. Anthropic's docs say Claude decides when to call a tool \"based on the user's request and the tool's description.\" The model \"only sees the schema you provided and the result you returned.\" Definitions also cost tokens on every request, because they are sent as part of the prompt, and the provider adds a few hundred tokens of its own instructions for tool use.</p>\n<h4 id=\"the-call-and-the-result\">The call and the result</h4>\n<p>When the model wants a tool, it stops with <code>stop_reason</code> set to <code>tool_use</code> and returns one or more <code>tool_use</code> blocks. Each block has an <code>id</code>, a <code>name</code> and an <code>input</code> object. Your code runs the function and sends a <code>tool_result</code> block in the next user message, naming the call through <code>tool_use_id</code>:</p>\n<pre class=\"shiki shiki-themes github-light github-dark\" style=\"background-color:#fff;--shiki-dark-bg:#24292e;color:#24292e;--shiki-dark:#e1e4e8\" tabindex=\"0\"><code><span class=\"line\"><span>{</span></span>\n<span class=\"line\"><span>  \"stop_reason\": \"tool_use\",</span></span>\n<span class=\"line\"><span>  \"content\": [</span></span>\n<span class=\"line\"><span>    {\"type\": \"text\", \"text\": \"I'll look up the order first.\"},</span></span>\n<span class=\"line\"><span>    {\"type\": \"tool_use\", \"id\": \"toolu_01Example1\",</span></span>\n<span class=\"line\"><span>     \"name\": \"lookup_order\", \"input\": {\"order_id\": \"A-1042\"}}</span></span>\n<span class=\"line\"><span>  ]</span></span>\n<span class=\"line\"><span>}</span></span></code></pre>\n<pre class=\"shiki shiki-themes github-light github-dark\" style=\"background-color:#fff;--shiki-dark-bg:#24292e;color:#24292e;--shiki-dark:#e1e4e8\" tabindex=\"0\"><code><span class=\"line\"><span>{</span></span>\n<span class=\"line\"><span>  \"role\": \"user\",</span></span>\n<span class=\"line\"><span>  \"content\": [</span></span>\n<span class=\"line\"><span>    {\"type\": \"tool_result\", \"tool_use_id\": \"toolu_01Example1\",</span></span>\n<span class=\"line\"><span>     \"content\": \"{\\\"status\\\": \\\"delivered\\\", \\\"total_cents\\\": 4599}\"}</span></span>\n<span class=\"line\"><span>  ]</span></span>\n<span class=\"line\"><span>}</span></span></code></pre>\n<p>Anthropic's docs add three rules. Send one result for every call in a parallel batch, all in one user message, with the result blocks before any text. Report failures with <code>is_error</code> set to <code>true</code>, and word the message so the model can act on it, such as \"Rate limit exceeded. Retry after 60 seconds.\" And return a result even for a call you decide not to run.</p>\n<p>OpenAI's function calling uses the same cycle with other names, as the table shows.</p>\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n<div class=\"table-wrap\"><table><thead><tr><th>Idea</th><th>Anthropic Messages API</th><th>OpenAI function calling</th></tr></thead><tbody><tr><td>Tool definition</td><td><code>name</code>, <code>description</code>, <code>input_schema</code></td><td><code>type</code>, <code>name</code>, <code>description</code>, <code>parameters</code></td></tr><tr><td>Model asks for a call</td><td><code>tool_use</code> block with <code>id</code>, <code>name</code>, <code>input</code> object</td><td><code>function_call</code> item with <code>call_id</code>, <code>name</code>, <code>arguments</code> string</td></tr><tr><td>Your answer</td><td><code>tool_result</code> block with <code>tool_use_id</code>, <code>is_error</code></td><td><code>function_call_output</code> item with <code>call_id</code>, <code>output</code> string</td></tr><tr><td>Enforce the schema</td><td><code>strict: true</code> on the tool</td><td><code>strict: true</code> on the function</td></tr></tbody></table></div>\n<p>Strict mode makes the arguments match the schema, so missing fields and wrong types go away. It cannot express a rule such as \"only if the order was delivered within the return window.\" The tool has to check business rules itself and refuse the call when they fail.</p>","headings":[{"id":"tool-calling","title":"Tool calling","depth":3},{"id":"the-call-and-the-result","title":"The call and the result","depth":4}]},{"id":"the-agent-loop","title":"The agent loop","html":"<h3 id=\"the-agent-loop\">The agent loop</h3>\n<p>The loop runs while the model keeps asking for tools. Anthropic's docs put it this way: while <code>stop_reason == \"tool_use\"</code>, execute the tools and continue the conversation. The loop stops on any other stop reason, such as <code>end_turn</code> for a final answer. Treat <code>max_tokens</code> as a failure, because a cut-off reply may hold half a tool call. Nothing in the loop caps how long it runs, so the harness needs its own limits:</p>\n<ul>\n<li>A <strong>step budget</strong>: the most model calls one run may make.</li>\n<li>A <strong>cost budget</strong>: the most tokens or money one run may spend.</li>\n<li>A <strong>stop condition</strong>: the model returns an answer with no tool calls.</li>\n<li>An <strong>escalation</strong>: a pause for a person before a risky action.</li>\n</ul>\n<p>The sketch makes these concrete. A scripted fake model asks for two tools, then answers. A registry maps each tool to a function and its required fields. The loop checks the step budget before each model call and the token budget after each reply, and it saves state before each tool runs. The four blocks form one standard-library file, <code>agent_loop.py</code>.</p>\n<pre class=\"shiki shiki-themes github-light github-dark\" style=\"background-color:#fff;--shiki-dark-bg:#24292e;color:#24292e;--shiki-dark:#e1e4e8\" tabindex=\"0\"><code><span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">import</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> json</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">import</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> os</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">import</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> uuid</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">TASK</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> =</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> \"A customer asks for a refund on order A-1042.\"</span></span>\n<span class=\"line\"><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">PAYMENTS</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> =</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> {}      </span><span style=\"color:#6A737D;--shiki-dark:#6A737D\"># stands in for the payments service: key -> stored result</span></span>\n<span class=\"line\"><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">REFUNDS_MADE</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> =</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> []  </span><span style=\"color:#6A737D;--shiki-dark:#6A737D\"># one entry per real refund, so we can count side effects</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">def</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\"> lookup_order</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(order_id: </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">str</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, key: </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">str</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">) -> </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">dict</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">:</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    return</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> {</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"order_id\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: order_id, </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"status\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"delivered\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"total_cents\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">4599</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">}</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">def</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\"> issue_refund</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(order_id: </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">str</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, amount_cents: </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">int</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, key: </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">str</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">) -> </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">dict</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">:</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    if</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> key </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">not</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> in</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> PAYMENTS</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">:            </span><span style=\"color:#6A737D;--shiki-dark:#6A737D\"># a repeated key returns the stored result</span></span>\n<span class=\"line\"><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">        REFUNDS_MADE</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">.append(key)       </span><span style=\"color:#6A737D;--shiki-dark:#6A737D\"># the real side effect happens once per key</span></span>\n<span class=\"line\"><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">        PAYMENTS</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">[key] </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> {</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"refunded_cents\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: amount_cents}</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    return</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> PAYMENTS</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">[key]</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#6A737D;--shiki-dark:#6A737D\"># The registry: the fields each tool needs, and the function that runs it.</span></span>\n<span class=\"line\"><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">TOOLS</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> =</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> {</span></span>\n<span class=\"line\"><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">    \"lookup_order\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: {</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"fn\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: lookup_order, </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"required\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: [</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"order_id\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">]},</span></span>\n<span class=\"line\"><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">    \"issue_refund\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: {</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"fn\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: issue_refund, </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"required\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: [</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"order_id\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"amount_cents\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">]},</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">}</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">def</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\"> fake_model</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(history: </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">list</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">) -> </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">dict</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">:</span></span>\n<span class=\"line\"><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">    \"\"\"Scripted stand-in for a model: two tool calls, then a final answer.\"\"\"</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">    done </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> sum</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">1</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> for</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> m </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">in</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> history </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">if</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> m[</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"role\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">] </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">==</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> \"tool\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">)</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    if</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> done </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">==</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 0</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">:</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">        call </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> {</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"id\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"call_1\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"name\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"lookup_order\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"input\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: {</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"order_id\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"A-1042\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">}}</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">        return</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> {</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"tool_calls\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: [call], </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"tokens\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">900</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">}</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    if</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> done </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">==</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 1</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">:</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">        call </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> {</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"id\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"call_2\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"name\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"issue_refund\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">,</span></span>\n<span class=\"line\"><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">                \"input\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: {</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"order_id\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"A-1042\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"amount_cents\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">4599</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">}}</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">        return</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> {</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"tool_calls\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: [call], </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"tokens\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">700</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">}</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    return</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> {</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"text\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"Refunded $45.99 for order A-1042.\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"tokens\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">300</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">}</span></span></code></pre>\n<pre class=\"shiki shiki-themes github-light github-dark\" style=\"background-color:#fff;--shiki-dark-bg:#24292e;color:#24292e;--shiki-dark:#e1e4e8\" tabindex=\"0\"><code><span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">def</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\"> execute</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(call: </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">dict</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, run_id: </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">str</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">) -> </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">dict</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">:</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">    entry </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> TOOLS</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">.get(call[</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"name\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">])</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    if</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> entry </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">is</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> None</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">:</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">        return</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> {</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"error\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">f</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"unknown tool </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">{</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">call[</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">'name'</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">]</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">}</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">}</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">    missing </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> [f </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">for</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> f </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">in</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> entry[</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"required\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">] </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">if</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> f </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">not</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> in</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> call[</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"input\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">]]</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    if</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> missing:                        </span><span style=\"color:#6A737D;--shiki-dark:#6A737D\"># the model sees this and can try again</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">        return</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> {</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"error\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">f</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"missing fields: </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">{</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">missing</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">}</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">}</span></span>\n<span class=\"line\"><span style=\"color:#6A737D;--shiki-dark:#6A737D\">    # The key comes from ids the harness recorded, never from the model's text.</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">    key </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> f</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">{</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">run_id</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">}</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">:</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">{</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">call[</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">'id'</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">]</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">}</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    try</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">:</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">        return</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> entry[</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"fn\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">](</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">**</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">call[</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"input\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">], </span><span style=\"color:#E36209;--shiki-dark:#FFAB70\">key</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">key)</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    except</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> Exception</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> as</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> exc:           </span><span style=\"color:#6A737D;--shiki-dark:#6A737D\"># report the failure, don't crash the run</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">        return</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> {</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"error\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">str</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(exc)}</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">CKPT</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> =</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> os.path.join(os.path.dirname(os.path.abspath(</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">__file__</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">)), </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"run_state.json\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">)</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">def</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\"> save</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(state: </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">dict</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">) -> </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">None</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">:</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    with</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> open</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">CKPT</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"w\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">) </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">as</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> f:</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">        json.dump(state, f)</span></span></code></pre>\n<pre class=\"shiki shiki-themes github-light github-dark\" style=\"background-color:#fff;--shiki-dark-bg:#24292e;color:#24292e;--shiki-dark:#e1e4e8\" tabindex=\"0\"><code><span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">def</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\"> run</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(max_steps: </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">int</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> =</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 6</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, token_budget: </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">int</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> =</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 5000</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, crash: </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">bool</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> =</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> False</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">) -> </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">str</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">:</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    if</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> os.path.exists(</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">CKPT</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">):           </span><span style=\"color:#6A737D;--shiki-dark:#6A737D\"># resume from the last checkpoint</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">        with</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> open</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">CKPT</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">) </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">as</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> f:</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">            state </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> json.load(f)</span></span>\n<span class=\"line\"><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">        print</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">f</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"resuming after </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">{</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">state[</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">'steps'</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">]</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">}</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> model call(s), \"</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">              f</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">{len</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(state[</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">'pending'</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">])</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">}</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> tool call(s) pending\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">)</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    else</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">:</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">        state </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> {</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"run_id\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">str</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(uuid.uuid4()), </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"steps\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">0</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"spent\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">0</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"pending\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: [],</span></span>\n<span class=\"line\"><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">                 \"messages\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: [{</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"role\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"user\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"content\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">TASK</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">}]}</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    while</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> True</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">:</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">        if</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> state[</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"pending\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">]:           </span><span style=\"color:#6A737D;--shiki-dark:#6A737D\"># finish calls recorded but not yet answered</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">            call </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> state[</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"pending\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">][</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">0</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">]</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">            result </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> execute(call, state[</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"run_id\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">])   </span><span style=\"color:#6A737D;--shiki-dark:#6A737D\"># the side effect happens here</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">            if</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> crash </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">and</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> call[</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"name\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">] </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">==</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> \"issue_refund\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">:</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">                raise</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> RuntimeError</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"crashed after the refund, before it was checkpointed\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">)</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">            state[</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"pending\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">].pop(</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">0</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">)</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">            state[</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"messages\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">].append(</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">                {</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"role\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"tool\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"tool_call_id\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: call[</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"id\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">], </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"content\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: result})</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">            save(state)</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">            continue</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">        if</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> state[</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"steps\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">] </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">>=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> max_steps:    </span><span style=\"color:#6A737D;--shiki-dark:#6A737D\"># step budget</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">            return</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> \"stopped: step budget reached\"</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">        reply </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> fake_model(state[</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"messages\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">])</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">        state[</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"steps\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">] </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">+=</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> 1</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">        state[</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"spent\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">] </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">+=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> reply[</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"tokens\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">]</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">        if</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> state[</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"spent\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">] </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">></span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> token_budget:  </span><span style=\"color:#6A737D;--shiki-dark:#6A737D\"># cost budget: this reply is not acted on</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">            return</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> \"stopped: token budget reached\"</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">        if</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> \"tool_calls\"</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> not</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> in</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> reply:      </span><span style=\"color:#6A737D;--shiki-dark:#6A737D\"># stop condition: a final answer, no tools</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">            return</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> reply[</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"text\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">]</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">        state[</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"messages\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">].append({</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"role\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"assistant\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"tool_calls\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: reply[</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"tool_calls\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">]})</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">        state[</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"pending\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">] </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> reply[</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"tool_calls\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">]</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">        save(state)                        </span><span style=\"color:#6A737D;--shiki-dark:#6A737D\"># checkpoint before any side effect runs</span></span></code></pre>\n<pre class=\"shiki shiki-themes github-light github-dark\" style=\"background-color:#fff;--shiki-dark-bg:#24292e;color:#24292e;--shiki-dark:#e1e4e8\" tabindex=\"0\"><code><span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">def</span><span style=\"color:#6F42C1;--shiki-dark:#B392F0\"> fresh</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">() -> </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">None</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">:</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    if</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> os.path.exists(</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">CKPT</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">):</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">        os.remove(</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">CKPT</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">)</span></span>\n<span class=\"line\"></span>\n<span class=\"line\"></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">if</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> __name__</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> ==</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\"> \"__main__\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">:</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    for</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> limits </span><span style=\"color:#D73A49;--shiki-dark:#F97583\">in</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> [{</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"max_steps\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">1</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">}, {</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"token_budget\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">: </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">1000</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">}]:</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">        fresh()</span></span>\n<span class=\"line\"><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">        print</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(limits, </span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"->\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, run(</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">**</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">limits))</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">    fresh()</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    try</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">:</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">        run(</span><span style=\"color:#E36209;--shiki-dark:#FFAB70\">crash</span><span style=\"color:#D73A49;--shiki-dark:#F97583\">=</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">True</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">)</span></span>\n<span class=\"line\"><span style=\"color:#D73A49;--shiki-dark:#F97583\">    except</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\"> RuntimeError</span><span style=\"color:#D73A49;--shiki-dark:#F97583\"> as</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\"> exc:</span></span>\n<span class=\"line\"><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">        print</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"crash:\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, exc)</span></span>\n<span class=\"line\"><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">    print</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"resume ->\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, run())</span></span>\n<span class=\"line\"><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">    print</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(</span><span style=\"color:#032F62;--shiki-dark:#9ECBFF\">\"refunds issued:\"</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">, </span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">len</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">(</span><span style=\"color:#005CC5;--shiki-dark:#79B8FF\">REFUNDS_MADE</span><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">))</span></span>\n<span class=\"line\"><span style=\"color:#24292E;--shiki-dark:#E1E4E8\">    fresh()</span></span></code></pre>\n<p>Running <code>python3 agent_loop.py</code> prints:</p>\n<pre class=\"shiki shiki-themes github-light github-dark\" style=\"background-color:#fff;--shiki-dark-bg:#24292e;color:#24292e;--shiki-dark:#e1e4e8\" tabindex=\"0\"><code><span class=\"line\"><span>{'max_steps': 1} -> stopped: step budget reached</span></span>\n<span class=\"line\"><span>{'token_budget': 1000} -> stopped: token budget reached</span></span>\n<span class=\"line\"><span>crash: crashed after the refund, before it was checkpointed</span></span>\n<span class=\"line\"><span>resuming after 2 model call(s), 1 tool call(s) pending</span></span>\n<span class=\"line\"><span>resume -> Refunded $45.99 for order A-1042.</span></span>\n<span class=\"line\"><span>refunds issued: 1</span></span></code></pre>\n<p>The first run stops at the step budget after one model call. The second run stops because the fake model's second reply would take the spend to 1,600 tokens, over the 1,000-token budget, so the loop never acts on that reply. The third run crashes just after the refund, before its result is saved. On restart, the saved state still lists that call as pending. The loop runs it again with the same key, the payments stub returns the stored result, and the run finishes with one refund.</p>\n<p>Two lines carry the design. The checkpoint saves the model's reply before any tool runs, and the key combines a run ID with a recorded call ID, so a retry sends the same key. If <code>issue_refund</code> ignores the key, the same crash test issues two refunds. The token check comes after the model call, so it limits waste but cannot prevent the first expensive call.</p>\n<h4 id=\"where-the-idea-comes-from\">Where the idea comes from</h4>\n<p>The 2022 ReAct paper by Yao and colleagues asked a model to generate \"reasoning traces and task-specific actions in an interleaved manner.\" Its reported gains, 34% on ALFWorld and 10% on WebShop in absolute success rate over imitation and reinforcement learning baselines, come from simulated environments.</p>","headings":[{"id":"the-agent-loop","title":"The agent loop","depth":3},{"id":"where-the-idea-comes-from","title":"Where the idea comes from","depth":4}]},{"id":"managing-context","title":"Managing context","html":"<h3 id=\"managing-context\">Managing context</h3>\n<p>Each step resends the whole history, so the context grows. Suppose the fixed part is 2,000 tokens and each step adds about 1,200. Step 10 reads 2,000 + 9 × 1,200 = 12,800 tokens. Across 10 steps the model reads 20,000 + 1,200 × 45 = 74,000 input tokens, which is 3.7 times what ten separate calls with the same prompt would read.</p>\n<p>The 70B model in the Llama 3 paper's Table 3 has 80 layers and 8 key/value heads. Its head dimension is 8,192 ÷ 64 = 128, so each token's BF16 KV cache is 2 × 80 × 8 × 128 × 2 = 327,680 bytes. Step 10 therefore needs 12,800 × 327,680 = 4,194,304,000 bytes, about 4.2 GB, or roughly three 4,096-token chats. Four 80 GB H100s, with an assumed 10% kept for runtime, leave 288 GB. After 140 GB of weights, 148 GB remains for KV cache, about 35 contexts of 12,800 tokens at once. <a href=\"/learn/inference-engines#prefix-caching\">Inference engines</a> explains how serving engines manage that memory.</p>\n<p>Anthropic's context engineering guide calls context \"a finite resource with diminishing marginal returns.\" Four techniques help:</p>\n<ul>\n<li><strong>Return less.</strong> The writing-tools guide advises tools to \"return only high signal information back to agents,\" and to page, filter or truncate large results.</li>\n<li><strong>Clear old results.</strong> Once a tool result has been used, the model rarely needs its full text again. Drop or shorten old results once the prompt passes a threshold, and keep the full history in your own storage. Anthropic's API offers this as a server-side option called context editing (in beta as of October 2026).</li>\n<li><strong>Summarize.</strong> Compaction, also in beta, replaces older turns with a summary the server writes. The docs say it helps because \"response quality degrades as a conversation grows.\"</li>\n<li><strong>Keep notes outside the window.</strong> The agent writes notes \"outside of the context window\" and reads them back when needed. Anthropic's memory tool offers this as files your application controls.</li>\n</ul>\n<h4 id=\"caching-the-stable-prefix\">Caching the stable prefix</h4>\n<p>Prompt caching matches prefixes: a change at or before a marked point misses the cache. As the conversation grows, the cache point moves forward, so each step reads most of its history from the cache. Keep the system prompt and tool definitions byte-for-byte stable, and only append messages. Cached input is much cheaper than fresh input: as of October 2026, Anthropic prices cache reads at a tenth of the base input price for most models. <a href=\"/learn/inference-engines#prefix-caching\">Inference engines</a> explains the same idea inside a self-hosted engine.</p>","headings":[{"id":"managing-context","title":"Managing context","depth":3},{"id":"caching-the-stable-prefix","title":"Caching the stable prefix","depth":4}]},{"id":"connecting-tools-with-mcp","title":"Connecting tools with MCP","html":"<h3 id=\"connecting-tools-with-mcp\">Connecting tools with MCP</h3>\n<p>Without a shared protocol, each agent application needs its own adapter for each system it touches, so N applications and M systems can mean N × M integrations. The Model Context Protocol (MCP) standardizes that boundary. The specification says it standardizes \"how to integrate additional context and tools into the ecosystem of AI applications.\"</p>\n<p>MCP has three roles. A <strong>host</strong> is the AI application. Each <strong>client</strong> inside the host talks to one <strong>server</strong>. Servers offer <strong>resources</strong> (context), <strong>prompts</strong> (templates) and <strong>tools</strong> (functions the model can run). Messages use JSON-RPC 2.0. The latest revision, dated 2026-07-28 in its URLs, describes requests as stateless, and it lists stdio and Streamable HTTP among its transports.</p>\n<figure class=\"diagram\" data-text=\"Agent host model + MCP client -> MCP server orders (tools/call); Agent host model + MCP client -> MCP server mail; Agent host model + MCP client -> MCP server files; MCP server orders -> Orders DB; MCP server mail -> Mail API; MCP server files -> File store\"><div class=\"diagram-scroll\"><svg class=\"dg\" viewBox=\"0 0 800 323\" width=\"800\" role=\"img\" aria-label=\"One host, several MCP servers. Each server wraps one system.\"><defs><marker id=\"dg-arrow\" viewBox=\"0 0 10 10\" refX=\"9\" refY=\"5\" markerWidth=\"7\" markerHeight=\"7\" orient=\"auto-start-reverse\"><path class=\"dg-arrowhead\" d=\"M0,1 L9,5 L0,9 z\"></path></marker></defs><line class=\"dg-edge\" x1=\"151\" y1=\"135.4\" x2=\"327\" y2=\"69.4\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge\" x1=\"151\" y1=\"162\" x2=\"327\" y2=\"162\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge\" x1=\"151\" y1=\"188.6\" x2=\"327\" y2=\"254.6\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge\" x1=\"471\" y1=\"42\" x2=\"647\" y2=\"42\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge\" x1=\"471\" y1=\"162\" x2=\"647\" y2=\"162\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge\" x1=\"471\" y1=\"282\" x2=\"647\" y2=\"282\" marker-end=\"url(#dg-arrow)\"></line><rect class=\"dg-node dg-box\" x=\"12\" y=\"136\" width=\"136\" height=\"52\" rx=\"8\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"80\" y=\"158.5\">Agent host</tspan><tspan x=\"80\" y=\"174.5\">model + MCP client</tspan></text><rect class=\"dg-node dg-box\" x=\"332\" y=\"16\" width=\"136\" height=\"52\" rx=\"8\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"400\" y=\"38.5\">MCP server</tspan><tspan x=\"400\" y=\"54.5\">orders</tspan></text><rect class=\"dg-node dg-box\" x=\"332\" y=\"136\" width=\"136\" height=\"52\" rx=\"8\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"400\" y=\"158.5\">MCP server</tspan><tspan x=\"400\" y=\"174.5\">mail</tspan></text><rect class=\"dg-node dg-box\" x=\"332\" y=\"256\" width=\"136\" height=\"52\" rx=\"8\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"400\" y=\"278.5\">MCP server</tspan><tspan x=\"400\" y=\"294.5\">files</tspan></text><path class=\"dg-node dg-store\" d=\"M652,20 a68,7 0 0,1 136,0 v44 a68,7 0 0,1 -136,0 z\"></path><path class=\"dg-line\" d=\"M652,20 a68,7 0 0,0 136,0\"></path><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"720\" y=\"49.5\">Orders DB</tspan></text><rect class=\"dg-node dg-box\" x=\"652\" y=\"138\" width=\"136\" height=\"48\" rx=\"8\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"720\" y=\"166.5\">Mail API</tspan></text><path class=\"dg-node dg-store\" d=\"M652,260 a68,7 0 0,1 136,0 v44 a68,7 0 0,1 -136,0 z\"></path><path class=\"dg-line\" d=\"M652,260 a68,7 0 0,0 136,0\"></path><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"720\" y=\"289.5\">File store</tspan></text><rect class=\"dg-edge-bg\" x=\"203\" y=\"93.4\" width=\"72\" height=\"18\" rx=\"4\"></rect><text class=\"dg-edge-label\" x=\"239\" y=\"106.4\" text-anchor=\"middle\">tools/call</text></svg></div><figcaption>One host, several MCP servers. Each server wraps one system.</figcaption></figure>\n<p>A host calls <code>tools/list</code> to discover tools. Each entry has a <code>name</code>, a <code>description</code>, an <code>inputSchema</code> in JSON Schema and optional <code>annotations</code>. The host runs one with <code>tools/call</code>. The spec separates two kinds of failure. A protocol error means a bad request, such as an unknown tool. A tool execution error comes back as a normal result with <code>isError</code> set to <code>true</code>, so the model can recover from it.</p>\n<p>The security section is blunt. Tools \"represent arbitrary code execution,\" hosts \"must obtain explicit user consent before invoking any tool,\" and clients must treat annotations as untrusted unless the server is trusted. MCP \"cannot enforce these security principles at the protocol level,\" so your application has to. MCP has no session that outlives a connection, so a stateful server returns a handle, such as a basket ID, that the model carries forward.</p>","headings":[{"id":"connecting-tools-with-mcp","title":"Connecting tools with MCP","depth":3}]},{"id":"running-agents-reliably","title":"Running agents reliably","html":"<h3 id=\"running-agents-reliably\">Running agents reliably</h3>\n<p>An agent run is a long workflow with side effects. Model calls fail, tools succeed and lose their replies, and workers crash. Three cases cover most of it.</p>\n<p><strong>A model call fails.</strong> Retry with backoff and jitter, as in <a href=\"/learn/reliability-patterns#retries-backoff-and-retry-storms\">Reliability patterns</a>. A model call changes nothing outside the program, so repeating it is safe, but each retry costs tokens.</p>\n<p><strong>A tool times out after succeeding.</strong> The agent cannot tell whether the refund happened, so use an <a href=\"/learn/reliability-patterns#idempotency-keys\">idempotency key</a>. Build it from ids the harness recorded before the call, as the sketch does. A random key per attempt makes every attempt look like a new request. A key derived from the model's wording can change between attempts.</p>\n<p><strong>The worker crashes between steps.</strong> Save the model's reply and its tool calls before any tool runs, then continue from the saved state on restart. Do not ask the model to redo completed steps, because a new call might choose differently.</p>\n<h4 id=\"checkpoints-and-durable-execution\">Checkpoints and durable execution</h4>\n<p>Saving state after each step is a <strong>checkpoint</strong>. Workflow engines go further. Temporal records each step as an event and rebuilds a workflow from that history. In its Go walkthrough, the service notices a dead worker after a Workflow Task Timeout, which defaults to 10 seconds. A new worker then re-runs the workflow code against the recorded history. Activity results are reused, and \"the Activity is not re-executed\" during replay. Its workflow code must be deterministic.</p>\n<p>For agents, put each model call and each tool call in its own activity, so replay reads the recorded reply instead of calling the model again. An engine cannot see inside an activity, though. After a crash mid-activity, it cannot tell whether the side effect happened, so the activity must be safe to repeat. Idempotency keys matter even inside a durable engine.</p>\n<p>If two workers could claim the same run, use a lease with a fencing token, as in <a href=\"/learn/consensus#leases-and-fencing-tokens\">Consensus</a>. A partly finished run, such as a refund followed by a failed confirmation email, needs compensating steps, as in <a href=\"/learn/distributed-transactions#sagas\">Distributed transactions</a>. Give each call its own timeout and the run a deadline, and trace every step, as covered in <a href=\"/learn/ai-evals-and-observability#tracing-model-calls\">Evaluating and observing AI systems</a>.</p>","headings":[{"id":"running-agents-reliably","title":"Running agents reliably","depth":3},{"id":"checkpoints-and-durable-execution","title":"Checkpoints and durable execution","depth":4}]},{"id":"permissions-and-sandboxes","title":"Permissions and sandboxes","html":"<h3 id=\"permissions-and-sandboxes\">Permissions and sandboxes</h3>\n<p>Give each tool only the power its task needs, enforced outside the model.</p>\n<ul>\n<li><strong>Scope each tool.</strong> <code>lookup_order</code> reads only the customer's orders. <code>issue_refund</code> refunds at most the order total. Each tool holds its own credentials, so the model never holds a key.</li>\n<li><strong>Enforce limits in the tool.</strong> A system prompt that says \"refunds over $100 need approval\" is a request. The payments service has to check it. The MCP spec says servers must validate all tool inputs and implement proper access controls.</li>\n<li><strong>Treat results as untrusted.</strong> Anthropic's docs warn that tool results \"often carry content from sources outside your control,\" and that they can carry instructions. Keep them inside tool result blocks. See <a href=\"/learn/ai-security#prompt-injection\">AI security</a>.</li>\n<li><strong>Ask a person first.</strong> The MCP spec says hosts \"must obtain explicit user consent before invoking any tool.\" Agent platforms usually offer three settings per tool: always allow, always ask (pause the run until someone approves), or decide per call by rule. Default tools you didn't write to always ask.</li>\n<li><strong>Sandbox code.</strong> Anthropic's code execution tool runs code \"in a secure, sandboxed environment,\" with internet access \"completely disabled for security.\" Copy those defaults: no network, time and memory limits, a fresh environment per run and no production credentials.</li>\n</ul>\n<p>An agent that reads untrusted text, can see private data and can send data outside is the most exposed combination. Remove one of the three where you can. See <a href=\"/learn/ai-security#limiting-what-agents-can-do\">AI security</a>.</p>","headings":[{"id":"permissions-and-sandboxes","title":"Permissions and sandboxes","depth":3}]},{"id":"multi-agent-systems","title":"Multi-agent systems","html":"<h3 id=\"multi-agent-systems\">Multi-agent systems</h3>\n<p>A multi-agent system runs several model loops on one task. In the common orchestrator and workers shape, a lead agent splits the task and gives each worker \"an objective, an output format, guidance on the tools and sources to use, and clear task boundaries.\" Each worker runs in its own context and returns a condensed result.</p>\n<figure class=\"diagram\" data-text=\"Task -> Lead agent keeps the plan; Lead agent keeps the plan <-> Worker 1 own context (brief, summary); Lead agent keeps the plan <-> Worker 2 own context (brief, summary); Lead agent keeps the plan <-> Worker 3 own context (brief, summary)\"><div class=\"diagram-scroll\"><svg class=\"dg\" viewBox=\"0 0 640 320\" width=\"640\" role=\"img\" aria-label=\"Orchestrator and workers. Each worker starts with a fresh context.\"><defs><marker id=\"dg-arrow\" viewBox=\"0 0 10 10\" refX=\"9\" refY=\"5\" markerWidth=\"7\" markerHeight=\"7\" orient=\"auto-start-reverse\"><path class=\"dg-arrowhead\" d=\"M0,1 L9,5 L0,9 z\"></path></marker></defs><line class=\"dg-edge\" x1=\"151\" y1=\"162\" x2=\"247\" y2=\"162\" marker-end=\"url(#dg-arrow)\"></line><line class=\"dg-edge\" x1=\"382\" y1=\"131\" x2=\"498\" y2=\"73\" marker-end=\"url(#dg-arrow)\" marker-start=\"url(#dg-arrow)\"></line><line class=\"dg-edge\" x1=\"393\" y1=\"162\" x2=\"487\" y2=\"162\" marker-end=\"url(#dg-arrow)\" marker-start=\"url(#dg-arrow)\"></line><line class=\"dg-edge\" x1=\"382\" y1=\"193\" x2=\"498\" y2=\"251\" marker-end=\"url(#dg-arrow)\" marker-start=\"url(#dg-arrow)\"></line><rect class=\"dg-node dg-client\" x=\"12\" y=\"138\" width=\"136\" height=\"48\" rx=\"24\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"80\" y=\"166.5\">Task</tspan></text><rect class=\"dg-node dg-box\" x=\"252\" y=\"136\" width=\"136\" height=\"52\" rx=\"8\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"320\" y=\"158.5\">Lead agent</tspan><tspan x=\"320\" y=\"174.5\">keeps the plan</tspan></text><rect class=\"dg-node dg-box\" x=\"492\" y=\"16\" width=\"136\" height=\"52\" rx=\"8\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"560\" y=\"38.5\">Worker 1</tspan><tspan x=\"560\" y=\"54.5\">own context</tspan></text><rect class=\"dg-node dg-box\" x=\"492\" y=\"136\" width=\"136\" height=\"52\" rx=\"8\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"560\" y=\"158.5\">Worker 2</tspan><tspan x=\"560\" y=\"174.5\">own context</tspan></text><rect class=\"dg-node dg-box\" x=\"492\" y=\"256\" width=\"136\" height=\"52\" rx=\"8\"></rect><text class=\"dg-label\" text-anchor=\"middle\"><tspan x=\"560\" y=\"278.5\">Worker 3</tspan><tspan x=\"560\" y=\"294.5\">own context</tspan></text><rect class=\"dg-edge-bg\" x=\"391.6\" y=\"93\" width=\"96.8\" height=\"18\" rx=\"4\"></rect><text class=\"dg-edge-label\" x=\"440\" y=\"106\" text-anchor=\"middle\">brief, summary</text><rect class=\"dg-edge-bg\" x=\"391.6\" y=\"114\" width=\"96.8\" height=\"18\" rx=\"4\"></rect><text class=\"dg-edge-label\" x=\"440\" y=\"127\" text-anchor=\"middle\">brief, summary</text><rect class=\"dg-edge-bg\" x=\"391.6\" y=\"213\" width=\"96.8\" height=\"18\" rx=\"4\"></rect><text class=\"dg-edge-label\" x=\"440\" y=\"226\" text-anchor=\"middle\">brief, summary</text></svg></div><figcaption>Orchestrator and workers. Each worker starts with a fresh context.</figcaption></figure>\n<p>Workers run in parallel and keep their contexts small. Anthropic's post says these systems suit tasks with \"heavy parallelization\" and information that \"exceeds single context windows.\" They also cost more. Anthropic reports that \"agents typically use about 4× more tokens than chat interactions,\" and that \"multi-agent systems use about 15× more tokens than chats.\" Its evaluation found that \"token usage by itself explains 80% of the variance,\" so on that evaluation, spending more tokens bought better results.</p>\n<p>Try the arithmetic with invented numbers. A chat answer uses 5,000 tokens, so the multiples give 20,000 tokens for an agent and 75,000 for a multi-agent run. At 10,000 tasks a day and an assumed $5 per million tokens (real prices vary), chat costs about $250 a day and multi-agent about $3,750. Per task, that is 2.5 cents against 37.5 cents.</p>\n<p>Coordination adds costs too. A weak brief wastes a worker's run, summaries drop detail, and a worker's output can carry injected instructions into the lead's context. The post says multi-agent systems are \"not a good fit\" for domains that require \"all agents to share the same context,\" and it notes that most coding tasks have fewer parallel parts than research. Start with one agent. Split work only when the parts are independent and the task is worth the tokens.</p>","headings":[{"id":"multi-agent-systems","title":"Multi-agent systems","depth":3}]},{"id":"measuring-agents","title":"Measuring agents","html":"<h3 id=\"measuring-agents\">Measuring agents</h3>\n<p>Measure runs by their final state and their cost.</p>\n<ul>\n<li><strong>Task success rate</strong>: the share of runs whose final state is correct, checked with code. For the refund agent, the ledger should show exactly one refund of $45.99 on order A-1042. A transcript that says \"refunded\" can still be wrong.</li>\n<li><strong>Cost per successful task</strong>: total spend divided by successes. Suppose 1,000 runs cost $0.10 each and 600 succeed. The total is $100, so each success costs about $0.17.</li>\n<li><strong>Steps per task</strong>: model calls per run, at the median and the 95th percentile. A rising count often signals a loop.</li>\n</ul>\n<p>A single-try success rate can hide inconsistency. The τ-bench paper defines pass^k as \"the chance that all k i.i.d. task trials are successful, averaged across tasks.\" Its best function-calling agent, using gpt-4o, averaged above 60% success per task, yet its pass^8 fell below 25%. If each try succeeds with probability 0.9 and the tries are independent, all eight succeed with probability 0.9^8, about 0.43.</p>\n<p>Build the test set from real requests, including tasks that need several tool calls. Anthropic's guide says prompts should be \"inspired by real-world uses,\" and that \"strong evaluation tasks might require multiple tool calls.\" Rerun the set after any change to the model, a prompt, a tool description or a limit. An agent that gets cheaper by succeeding less has saved nothing. The <a href=\"/learn/ai-evals-and-observability#building-an-eval-set\">evals chapter</a> explains how to build the set.</p>","headings":[{"id":"measuring-agents","title":"Measuring agents","depth":3}]},{"id":"check-yourself","title":"Check yourself","html":"<h3 id=\"check-yourself\">Check yourself</h3>\n<details>\n<summary>An agent takes 8 steps. The fixed prompt is 3,000 tokens, and each step adds 1,500 tokens of tool results and 300 tokens of model output. How many input tokens does step 8 read, and how many do all 8 steps read together?</summary>\n<p>Step 8 reads 3,000 + 7 × 1,800 = 15,600 tokens. Over all 8 steps, the fixed 3,000 is read 8 times (24,000), and the growth adds 1,800 × 28 = 50,400, for 74,400 input tokens.</p>\n</details>\n<details>\n<summary>A refund call times out after the payments service processed it. The harness retries with a new random key. What happens, and what should the key be?</summary>\n<p>The service treats the retry as a new request, so the customer is refunded twice. The key should come from the run ID and the recorded call ID, so every attempt sends the same key and gets the stored result. Retry from the saved call record, since asking the model again may produce a different call.</p>\n</details>\n<details>\n<summary>After a release, cost per task rose from $0.12 to $0.31, success stayed at 90%, and steps per task rose from 6 to 14. Where do you look first?</summary>\n<p>Look at the traces of the longer runs. A rising step count with flat success usually means a loop: one tool called repeatedly, or errors that do not say what to try next. Check whether tool results grew and whether a changed description sent the model to the wrong tool.</p>\n</details>\n<details>\n<summary>Every night a job matches 20,000 bank lines to invoices with fixed rules and sends unmatched lines to a person. Should it be a single model call, a workflow or an agent?</summary>\n<p>A workflow. The steps are known, so ordinary code does the matching. A model can help with messy free-text references on unmatched lines, and anything it cannot resolve goes to a person.</p>\n</details>\n<details>\n<summary>An MCP server you did not write marks a tool readOnlyHint: true. A teammate wants to auto-approve every read-only tool from any server. What do you say?</summary>\n<p>The spec says clients must treat annotations as untrusted unless the server is trusted, so the hint is only the server's claim. Approve tools by how far you trust the server, and keep human approval for anything that can write or send data.</p>\n</details>\n<details>\n<summary>An agent is correct on 85% of independent tries. What is the chance it is correct on all 5, and what would you change before using it in a pipeline that cannot retry?</summary>\n<p>0.85 × 0.85 × 0.85 × 0.85 × 0.85 is about 0.44 (pass^5), so a pipeline that needs all five to pass fails about 56% of the time. Raise per-try reliability with better tools and final-state checks, add idempotent retries, or send failures to a person.</p>\n</details>\n<h3 id=\"further-reading\">Further reading</h3>\n<ul>\n<li><a href=\"https://modelcontextprotocol.io/specification/latest\" target=\"_blank\" rel=\"noopener noreferrer\">Model Context Protocol specification</a>: roles, base protocol and security principles, in the 2026-07-28 revision.</li>\n<li><a href=\"https://modelcontextprotocol.io/specification/2026-07-28/server/tools\" target=\"_blank\" rel=\"noopener noreferrer\">MCP tools specification</a>: exact <code>tools/list</code> and <code>tools/call</code> shapes and error types.</li>\n<li><a href=\"https://platform.claude.com/docs/en/agents-and-tools/tool-use/overview\" target=\"_blank\" rel=\"noopener noreferrer\">Tool use with Claude</a>: client and server tools, with the tool-use token table.</li>\n<li><a href=\"https://platform.claude.com/docs/en/agents-and-tools/tool-use/handle-tool-calls\" target=\"_blank\" rel=\"noopener noreferrer\">Handle tool calls</a>: the <code>tool_use</code> and <code>tool_result</code> formats, <code>is_error</code> and untrusted results.</li>\n<li><a href=\"https://platform.claude.com/docs/en/agents-and-tools/tool-use/parallel-tool-use\" target=\"_blank\" rel=\"noopener noreferrer\">Parallel tool use</a>: running several calls at once.</li>\n<li><a href=\"https://platform.claude.com/docs/en/agents-and-tools/tool-use/how-tool-use-works\" target=\"_blank\" rel=\"noopener noreferrer\">How tool use works</a>: where each kind of tool runs, and the loop's stop reasons.</li>\n<li><a href=\"https://developers.openai.com/api/docs/guides/function-calling\" target=\"_blank\" rel=\"noopener noreferrer\">OpenAI function calling guide</a>: the same cycle in OpenAI's shapes.</li>\n<li><a href=\"https://www.anthropic.com/research/building-effective-agents\" target=\"_blank\" rel=\"noopener noreferrer\">Building effective agents</a> (Anthropic, December 19, 2024): workflows versus agents. Its page notes that much of the tooling has changed.</li>\n<li><a href=\"https://arxiv.org/abs/2210.03629\" target=\"_blank\" rel=\"noopener noreferrer\">ReAct: Synergizing Reasoning and Acting in Language Models</a> (Yao et al., 2022): interleaving reasoning and actions.</li>\n<li><a href=\"https://www.anthropic.com/engineering/multi-agent-research-system\" target=\"_blank\" rel=\"noopener noreferrer\">How we built our multi-agent research system</a> (Anthropic, June 13, 2025): token multiples and the lead-and-subagent design.</li>\n<li><a href=\"https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents\" target=\"_blank\" rel=\"noopener noreferrer\">Effective context engineering for AI agents</a> (Anthropic, September 29, 2025): compaction, note-taking and sub-agents.</li>\n<li><a href=\"https://www.anthropic.com/engineering/writing-tools-for-agents\" target=\"_blank\" rel=\"noopener noreferrer\">Writing effective tools for agents</a> (Anthropic, September 11, 2025): what tools should return.</li>\n<li><a href=\"https://platform.claude.com/docs/en/build-with-claude/prompt-caching\" target=\"_blank\" rel=\"noopener noreferrer\">Prompt caching</a>: prefix matching and cache reads.</li>\n<li><a href=\"https://platform.claude.com/docs/en/build-with-claude/compaction\" target=\"_blank\" rel=\"noopener noreferrer\">Compaction</a> and <a href=\"https://platform.claude.com/docs/en/build-with-claude/context-editing\" target=\"_blank\" rel=\"noopener noreferrer\">Context editing</a>: two server-side ways to shrink context, both in beta.</li>\n<li><a href=\"https://platform.claude.com/docs/en/agents-and-tools/tool-use/code-execution-tool\" target=\"_blank\" rel=\"noopener noreferrer\">Code execution tool</a>: a sandbox with no internet access.</li>\n<li><a href=\"https://platform.claude.com/docs/en/managed-agents/permission-policies\" target=\"_blank\" rel=\"noopener noreferrer\">Permission policies</a>: one platform's approval modes for tool calls, in beta.</li>\n<li><a href=\"https://docs.temporal.io/encyclopedia/event-history/event-history-go\" target=\"_blank\" rel=\"noopener noreferrer\">Event History walkthrough with the Go SDK</a> (Temporal): how durable execution rebuilds state from recorded events.</li>\n<li><a href=\"https://arxiv.org/html/2407.21783\" target=\"_blank\" rel=\"noopener noreferrer\">The Llama 3 Herd of Models</a> (Meta, 2024): Table 3 gives the layer and head counts behind the KV-cache arithmetic.</li>\n<li><a href=\"https://www.nvidia.com/en-us/data-center/h100/\" target=\"_blank\" rel=\"noopener noreferrer\">NVIDIA H100</a>: the 80 GB memory figure for the SXM part.</li>\n<li><a href=\"https://arxiv.org/abs/2406.12045\" target=\"_blank\" rel=\"noopener noreferrer\">τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains</a> (Yao et al., 2024): the pass^k reliability metric and its retail results.</li>\n</ul>","headings":[{"id":"check-yourself","title":"Check yourself","depth":3},{"id":"further-reading","title":"Further reading","depth":3}]}]}