# Prompt Caching in Agents Earendil's post argues that for a coding agent, prompt caching is not an optimization sitting below the product but a constraint shaping it. The starting observation is that an LLM only looks like a function from text to text. A coding agent sends the system prompt, tool definitions, project instructions, conversation history, tool calls and tool results, then on the next turn sends nearly all of it again plus a little new material. Once a session reaches hundreds of thousands of tokens, whether the provider can skip re-reading that prefix decides latency, price, which tools you dare load, and which product features are affordable at all. The examples throughout are from [[pi-coding-agent]], and the cache-statistics code the post links sits at `earendil-works/pi` rather than the `badlogic/pi-mono` path the toolbox page records. ## What the cache holds A transformer handles a request in two phases: **prefill**, where it reads the input tokens and computes attention state, and **decode**, where it emits new tokens one at a time. At each attention layer every processed token yields a key and a value, both arrays of numbers rather than hash-table entries. A new token's *query* is compared against earlier *keys* to score how relevant each earlier token is, and those scores form a weighted mixture of the corresponding *values*. Retaining those keys and values so later tokens can attend without recomputation is the KV cache; prompt caching extends its lifetime past a single generation. The property that matters is that the state corresponds to a specific token prefix. Two prompts that mean the same thing but tokenize differently share nothing. Change a token in the middle and everything after it is a different continuation. [[kv-cache-sizing]] gives the arithmetic for how much memory that state costs per token; the post's own perspective note is that with the usual tricks a long conversation's cache comes down to a handful of gigabytes, smaller than most people assume. ## Where it lives, and why that leaks into product behavior Two broad implementations. **Session affinity** keeps the cache on or near the GPU that computed it and routes the next request back to the same worker, so a session ID or cache key works as a routing hint — cheap enough to handle at the HTTP load balancer without inspecting the payload. Nothing large moves over the network, but scheduling is constrained: that worker can be overloaded, restart, evict the entry, or lose out to a router that decides fleet balance matters more than one session's cache. The alternative is to **distribute the cache**, storing KV blocks in another memory tier or across workers so a request is not pinned to one GPU. That buys scheduling flexibility and recovery at the price of indexing, moving and evicting the blocks. The affinity variant is the same addressing problem [[stateful-agent-routing]] describes: you need a name that reaches one particular process holding state, not just any healthy backend. Pi sessions are trees rather than lists, and that interacts badly with both. `/tree` moves the active conversation to an earlier point and continues down another branch; a rewind drops the active suffix without deleting it from the session file. All branches share one session ID, so the router sees one session while the cache sees three token sequences with partial overlap. Jumping to a sibling branch may still reuse the shared root if the provider keeps reusable prefix blocks, or may reuse almost nothing if it only retains the hottest continuation. `/fork` produces the mirror-image failure: a new session ID carrying nearly identical context, which a cache isolated by session key never notices is reusable. Session identity only helps the infrastructure guess where to look. The reusable prefix is what decides what can be skipped. Provider APIs expose this in two styles. Anthropic's traditional interface takes explicit `cache_control` breakpoints after stable regions — system prompt, tool definitions, latest cacheable content — with explicit pricing to match: you pay for cache writes and choose a retention tier. Automatic prefix caching asks for none of that and finds the reusable prefix itself, with any cache key acting as a routing hint rather than a promise. ## Lazy tool loading is a cache bug Tool definitions are folded into the system prompt before the conversation, so their names, descriptions and JSON schemas are ordinary model input sitting at the very front of the request. Add one tool, remove one, change a schema, or serialize the same set in a different order, and the first mismatch moves to near the start of the prompt: ``` turn 1: [system][read][write][bash][conversation...........] turn 2: [system][read][write][bash][deploy][conversation...] | old conversation is now after a mismatch ``` This is the trap in plugin systems and MCP-style tool catalogs. Loading a tool only when it becomes relevant reads as efficient because fewer schemas go out initially, and on most models it invalidates the entire conversation that follows, so saving a few hundred schema tokens costs a re-prefill of tens of thousands. It is a concrete cost to set against the "MCP for service access" side of [[mcp-vs-skills]]: a large catalog is expensive once, but a catalog that changes shape mid-session is expensive every time it changes. Newer model APIs offer **additive tool loading**, where a tool becomes available at a specific tool result inside the transcript instead of being spliced into the original tool list, leaving the prefix untouched. Pi supports this where the model does: an extension making a purely additive change through `setActiveTools()` gets the added names recorded on the tool result, delivered as deferred definitions plus a `tool_reference` on supported Anthropic models and as tool-search items on supported OpenAI ones. Everything else falls back to sending the complete active tool list next request, which is correct and may wipe the cache. Additive is the operative word. Removing tools, swapping one loadout for another, rebuilding the system prompt, shuffling tool order or injecting a timestamp all rewrite earlier input. Since extensions can do any of those, Pi can offer cache-friendly mechanisms but cannot guarantee cache stability on their behalf, and the post's observation is that most extensions treat cache efficiency as an afterthought — partly because on a fixed subscription the cost of a miss is invisible to the person writing the extension. ## TTLs and the price of a miss Anthropic's default five-minute cache is shorter than ordinary coding activity, and the provider sees a sequence of isolated requests where the user sees one continuous session: ``` model request --> run tests for 7 minutes --> model request no cache traffic here ``` A long build, a test suite, lunch, or stopping to read a diff outlives the entry. Pi follows the five-minute default because it is not a permitted harness on Anthropic's subscription, and the post notes from reading Claude Code's own code that Anthropic raises that timeout to an hour for their subscription users — worth it there, often not worth it at API token prices. Users on direct APIs can set `PI_CACHE_RETENTION=long` to request longer retention, which is a request and not a guarantee: Pi cannot pick an eviction policy, keep a GPU alive, or hold a cache open while no request is in flight. The bill follows from the pricing split between uncached input, cache writes and cache reads. On 100,000 tokens of history plus a short new message, a hit charges nearly all of it at the discounted read price; a miss reprocesses the whole history at the regular input price and may charge to write it back. That is why typing `continue` after a coffee can cost more than the answer it produces, and why in a long session re-reading old input dominates generating new output. Incentives do not all point the same way. Users want hits for latency and price, and an operator that owns the GPUs wants them too, since less prefill means more requests per unit of hardware. A gateway or reseller billing input tokens at the uncached rate can earn more revenue from a miss, with the party that controls routing not being the party that pays for it. The post stops short of alleging sabotage and lands on the useful version of the claim: cache performance should be observable rather than inferred from a surprising invoice. It also notes the honest tension — strict cache adherence costs a router the freedom to move you to a cheaper or better backend mid-session, since KV state does not travel. Anything that shifts requests between providers or models, which is the entire premise of [[claude-code-router]] and [[cursor-bridge]], trades cache hits for routing freedom. It is the same measurement gap [[ai-token-budget-explosion]] describes at the enterprise level, where spend is visible and per-task cost is not. ## Why Pi does not prune Deleting old tool results to control cost changes the prefix at the deletion point, so everything surviving after it may be re-prefilled. The post gives the break-even directly: ``` one-time rewrite cost ~= surviving tokens after the edit * (uncached price - cache-read price) future savings per turn ~= pruned tokens * cache-read price ``` Rewriting a long cached context can cost more immediately than the pruning saves over the rest of the session. There is a behavioral argument on top of the accounting one: old tool results hold the evidence the model used for later decisions, and a summary that preserves the gist can still degrade behavior. So Pi keeps a stable append-oriented transcript and reserves compaction for real context pressure, counting it as a cache reset rather than a cache failure in session statistics, because it deliberately creates new context instead of accidentally re-billing unchanged context. Pruning still wins in one case: a provider that gives no cache-read discount, or a setup that cannot reach good hit rates anyway, where a shorter prompt at least lets the router balance freely. The goal is stated as the trade-off, not the minimum — context, reuse, latency and price together. This is the cache-aware counterweight to the compaction-and-summarize instinct in [[context-engineering]]. ## Making misses visible What Pi claims it can do is keep stable inputs stable and report the result. The interactive footer carries cumulative cache reads and writes as `R` and `W` plus `CH` for the last request's hit rate, and `/session` prints the full picture, including an estimate of what significant misses re-billed: ``` Tokens Input: 7,129,883 Cached: 6,776,832 (95.0%) Uncached: 353,051 Output: 30,013 Cost Total: $6.054 Cache Re-billed: $0.728 (161,744 tokens, 2 misses) ``` Two misses out of a 178-message session cost twelve percent of the session's total. Enabling `showCacheMissNotices` in settings makes Pi post a warning inline after a significant miss with the estimated re-billed tokens and dollars, naming a model switch or an idle gap when it can observe one and otherwise reporting the miss without guessing at the provider's internals. For Claude Code, [[toolbox/tare|tare]] does a similar accounting after the fact from local session logs, charging each tool for the context it caused to be re-sent rather than for what it returned. The post's closing list of usual suspects is worth keeping as a checklist, since each one maps to a mechanism above: idling past the retention window; switching model or provider, since KV state is model-specific and does not move; branch navigation through `/tree`, rewinds and forks; compaction or manual history rewriting; tool or reasoning-level changes; dynamic system prompts carrying timestamps or rotating project context; extensions that transform old messages on their way out; and provider-side routing or eviction, where the prompt is byte-identical and the blocks simply are not where the request landed.