--- source: 'https://howaiworks.ai/glossary/prompt-caching' section: glossary title: Prompt Caching description: >- Repeated prompt prefixes bill at a fraction of the input price. How prefix matching works, what silently breaks it, and where the break-even sits. tags: - Inference - LLM - AI Infrastructure - Efficiency - Prompt Engineering - Context Window category: Infrastructure & Hardware datePublished: '2026-07-24' lastUpdated: '2026-07-24' --- # Prompt Caching > Repeated prompt prefixes bill at a fraction of the input price. How prefix matching works, what silently breaks it, and where the break-even sits. ## Definition **Prompt caching bills the front of your prompt at a fraction of its normal price, provided the bytes have not changed by so much as one character.** Large language model APIs are stateless: every request resends the whole conversation, and a coding agent on its fortieth turn is paying to re-read the same 30,000-token system prompt and tool list for the fortieth time. Prompt caching lets the provider keep the processed form of that repeated prefix, and charge you for storage-plus-reuse instead of re-reading. On Anthropic's published multipliers as of July 2026, a cache read costs **0.1x** the base input price and a cache write **1.25x** — so a 20,000-token prefix reused across a thousand calls drops from **$100.00 to $10.12**, a **89.9%** saving. The catch, and the reason most teams who switch it on see nothing, is the qualifier. Caching is a **prefix match**: the provider hashes your prompt from byte zero, and a single differing character at position N destroys the match for everything after N. Interpolate `datetime.now()` into the top of your system prompt and you have not merely lost the discount — you now pay the **1.25x write premium on every single call and never once read**, which is 25% *more* than never caching at all. Nothing errors. The bill just goes up. ## How It Works ### Everything follows from the prefix match There is exactly one rule on this page, and every practical consequence is derived from it. The cache key is a **cumulative hash of the rendered prompt** up to the point you mark. Block five's hash covers blocks one through five. So the question a provider asks is never "have I seen this system prompt before?" — it is "have I seen this *exact sequence of bytes from the beginning* before?" That reframing explains behaviour that otherwise looks arbitrary: **A timestamp anywhere in the prefix invalidates everything behind it.** Not just the line it sits on. If the current date is rendered into the system prompt at position 1 of 3, then blocks 2 and 3 are unreachable, even though they are byte-identical to last time. The 20,000 tokens of frozen instructions sitting *after* the timestamp are re-read at full price on every request, forever. **Non-deterministic serialisation does the same thing, invisibly.** `json.dumps()` on a Python dict emits keys in insertion order; build the dict by a different code path and the same data serialises to different bytes. Iterating a `set` is worse — CPython randomises string hashing per process, so your tool list can serialise differently after a restart. The prompt is semantically identical, the hash is not, and nothing in your logs will say so. **Changing the tool list invalidates the entire conversation.** Providers render the prompt in a fixed order — Anthropic documents it as `tools`, then `system`, then `messages` — which puts tool definitions at position zero. Adding one tool to an [agent](https://howaiworks.ai/glossary/ai-agent) mid-session does not invalidate "the tools section". It invalidates the tools, the system prompt, and every turn of the conversation built on top of them. This is why "modes" that swap the [tool set](https://howaiworks.ai/glossary/function-calling) per request are a caching disaster, and why building tools per-user means no two users ever share a cache entry. **So stable content must physically precede volatile content.** That is the whole design rule. Frozen system prompt and deterministic tool list first, then slowly-changing context (retrieved documents, few-shot examples), then the per-request material — the user's question, the timestamp, the session ID — last. Get the ordering right and caching mostly works for free; get it wrong and no amount of configuration will rescue it. ### The breakpoint, and why writes only happen where you mark Marking a block tells the provider to write **one** cache entry: the hash of the prefix ending at that block. It does not write entries at earlier positions. On the next request the provider computes the hash at your mark, and if it misses, walks backward looking for an entry an *earlier request* wrote — Anthropic caps that walk at **20 blocks**. The consequence trips people up. If you mark the last block of a request whose last block is the user's question, you write a distinct entry every time and read none of them, because the lookback finds no prior write behind it. It is not searching for stable content; it is searching for previous writes. The mark belongs on the last *stable* block, not the last block. The 20-block limit has its own sharp edge in long agent runs. If a single turn appends more than 20 blocks — easy when a turn fires ten tool calls and receives ten results — the next request's lookback never reaches the previous turn's entry, and a conversation that was hitting cache all morning quietly stops. Anthropic allows up to **4 breakpoints** per request precisely so you can plant an intermediate mark inside long turns. ### Deriving the break-even Take one unit as the cost of processing the prefix at the base input price. Over N requests: ``` uncached = N cached = w + r x (N - 1) w = write multiplier, r = read multiplier ``` They are equal when `N = (w - r) / (1 - r)`. With Anthropic's `w = 1.25` and `r = 0.10`, that is **1.28 requests** — meaning the second request is already cheaper, and the first one alone costs you 25% extra. Choose the longer-lived cache, written at `w = 2`, and break-even moves to **2.11**: the second request is a small loss and the third pays. That is the real trade in TTL choice. A longer-lived entry survives gaps in bursty traffic, but the doubled write means it needs more reads to earn its keep. The ceiling is worth naming too. As N grows, `cached / uncached` approaches `r`, so the saving asymptotes at **90%** and never exceeds it. Prompt caching does not make the prefix free; it makes it a tenth. ### Where a template edit actually costs you Both [MLOps](https://howaiworks.ai/glossary/mlops) and [monitoring](https://howaiworks.ai/glossary/monitoring) practice warn that a prompt template edit adding 400 tokens raises the bill on every call, forever, and breaks no test. Caching changes that warning in a way that is worth internalising: **with a cache in play, where the tokens go matters roughly ten times more than how many there are.** Take that 20,000-token prefix on an Opus-tier model at $5 per million input tokens (July 2026), reused often enough to sit at steady state. The cached read costs **$0.0100** per call. Now add the same 400 tokens three different ways: - **Static, above the breakpoint.** The prefix becomes 20,400 tokens and still caches. Per call: **$0.0102** — a 2% rise. - **Static, below the breakpoint.** Those tokens never cache, so they bill at the full base rate on every call. Per call: **$0.0120** — a 20% rise, ten times worse than the identical bytes placed above the mark. - **Volatile, above the breakpoint** (a rendered timestamp, a request ID). Nothing matches, so every call is a fresh 20,400-token write at 1.25x. Per call: **$0.1275** — **12.75x** the baseline. Same 400 tokens; +2%, +20%, or +1,175% depending only on position. The reviewable artefact is no longer the diff's size, it is where in the prompt the diff landed. ### It is not the same thing as the KV cache These two get conflated constantly, and the distinction is clean. The [KV cache](https://howaiworks.ai/glossary/kv-cache) is the *within-request* mechanism: as the model reads, each layer stores a key and a value vector per token so later tokens can attend to the past without recomputing it. It is why generating the second word of an answer is cheap. Prompt caching is the *across-request product* built on that mechanism: the provider keeps those keys and values resident between your API calls and sells you the right to skip the read-in phase. Same physics, different billing surface — and only one of them is something you can break with a `datetime.now()`. ## Real-World Applications **Coding agents and long-running assistants.** This is where caching stopped being an optimisation and became a precondition. An [agentic workflow](https://howaiworks.ai/glossary/agentic-workflow) that reads a repository and then takes forty turns resends the entire accumulated transcript on every turn; at turn forty the prompt is mostly things the model has already been shown thirty-nine times. Without caching the cost of a session grows with the *square* of its length, since each of N turns resends O(N) of history. With caching, the repeated part bills at a tenth and the growth curve flattens to something a product can be priced on. It is not a coincidence that per-turn pricing for coding assistants arrived at the same time as prompt caching did. **Retrieval over a corpus that changes slower than the questions.** [RAG](https://howaiworks.ai/glossary/retrieval-augmented-generation) systems answering many questions against the same policy manual, contract set or knowledge base can put the retrieved documents above the breakpoint and the question below it. The ordering is the whole design decision, and it is the opposite of what a naive implementation does — most templates put retrieved chunks *last*, right before the answer, because that reads better. Reading better costs 10x. **Classification and extraction at volume.** A pipeline that runs the same 8,000-token instruction block plus twenty few-shot examples across a million documents is the ideal shape: an enormous fixed prefix, a tiny variable suffix. Break-even arrives on the second document and the remaining 999,998 run at a tenth of the input price. **Deciding whether a longer prompt is affordable.** Caching changes the answer to "can we afford to put the whole style guide in the system prompt?" from *no* to *usually*, provided the guide is static. That is a genuine capability shift, not just a discount — the constraint on system-prompt size moves from cost toward the [context window](https://howaiworks.ai/glossary/context-window) itself. ## Key Concepts **Cache write, cache read, and uncached input** are three different line items with three different prices, and every provider that supports caching reports them as separate counters in the response. Anthropic's are `cache_creation_input_tokens`, `cache_read_input_tokens` and `input_tokens`, and the relationship worth memorising is that they *sum* to the total prompt: ``` total input = cache_read + cache_creation + input_tokens ``` `input_tokens` is only the remainder after your last breakpoint, which is why an agent that ran all morning can show a suspiciously small `input_tokens` — the rest was served from cache. Reading that field alone as "how big was my prompt" is a reliable way to misjudge both cost and rate limits. **TTL** is how long an unused entry survives. Anthropic's default is five minutes, refreshed at no charge every time the entry is read, with a longer one-hour option at the doubled write price. The refresh-on-read behaviour matters more than the raw number: traffic arriving more often than the TTL keeps the cache alive indefinitely at no extra cost, so the longer TTL only earns its premium for genuinely bursty workloads with idle gaps. **Minimum cacheable length** is a floor below which nothing caches at all. Anthropic's published minimums as of July 2026 range from 512 to 4,096 tokens depending on the model; Google documents a per-model minimum for Gemini's implicit context caching too. It is not a quirk of one vendor — it exists because bookkeeping for a tiny entry costs more than the entry saves. ## Challenges **The silent failure: the cache never populates.** Your prefix is below the minimum cacheable length for the model you are calling. There is no error, no warning, and no field that says "too short" — the request is simply processed without caching. The tell is that *both* the write and read counters come back zero, request after request. This bites hardest when a prompt sits just under the threshold, and the fix is counter-intuitive: make the cached section *longer* until it clears the floor, since a read is so much cheaper than uncached input that padding up to the minimum can lower the bill. It also bites when you change models — the minimum is per-model, so a prefix that cached happily on one model can silently stop on another, with nothing in the diff to blame. **The invisible failure: you write forever and never read.** Something in the prefix varies per request, so every call writes a fresh entry and no call ever matches one. This is strictly worse than not caching: you pay 1.25x base input on every request instead of 1.00x, a permanent **25% surcharge**, while your configuration says caching is enabled and your dashboards look fine. The diagnostic is one line — if `cache_creation_input_tokens` is non-zero on every request and `cache_read_input_tokens` is never non-zero, hunt the invalidator. Candidates, in rough order of frequency: a rendered timestamp or date, a UUID or request ID, a user or session identifier interpolated into the system prompt, unsorted JSON, a conditionally-assembled system prompt where each feature-flag combination is a distinct prefix, and a tool list built per-user. **Forks and sub-agents rebuild the prefix and lose it.** A summarisation pass, a compaction step or a spawned sub-agent typically constructs its own request. If it rebuilds the system prompt or the tool list with *any* difference — a different assembly order, a trimmed instruction, a different model — it misses the parent's cache entirely and pays a full cold write. The fix is unglamorous: copy the parent's prefix verbatim and append the fork-specific material at the end. **Parallel requests all miss.** An entry only becomes readable once the first response begins, so firing N identical-prefix requests simultaneously means N cache writes and zero reads — the expensive outcome, arrived at by doing the obviously efficient thing. Fan-out patterns need to send one request, wait for it to start returning, then release the rest. **Cheap tokens are still tokens.** Cached input costs a tenth but occupies the [context window](https://howaiworks.ai/glossary/context-window) in full, counts against rate limits, and does nothing for generation speed — caching cuts time to first token, not the per-token decode that dominates a long answer. Teams that discover the discount sometimes respond by stuffing the prefix with everything that might conceivably help, and rediscover that a bloated context degrades answer quality at a tenth of the price. ## Future Trends **Automatic caching is absorbing the breakpoint.** Providers increasingly place the mark for you — Google's Gemini caching is implicit by default, and Anthropic offers a top-level automatic mode that moves the breakpoint forward as a conversation grows. This removes the most common mistake and introduces a subtler one: automatic placement puts the mark on the last cacheable block, which in a prompt with a varying suffix is precisely the block that changes every request. The failure mode migrates from "I forgot to mark anything" to "it marked the wrong thing and I did not notice." **Prompt layout is becoming a reviewed artefact.** Once the difference between a 2% and a 1,175% cost change is *where* a string was inserted, prompt assembly stops being template code and starts being something with an invariant worth testing — a unit test that renders the prefix twice and asserts the bytes match is trivial to write and catches every invalidator on the list above. Expect that assertion to become as routine in LLM services as a schema check. **The API surface is growing affordances specifically to protect the prefix.** The clearest example is mid-conversation system messages: instead of editing the top-level system prompt when an operator instruction arrives mid-session — which changes byte zero and re-processes the entire conversation — providers now let you append a system-role message *after* the cached history. Watch for more of this shape: features whose entire purpose is to let something change without changing the front of the prompt. **Cache-aware routing.** In self-hosted serving, the natural next step is a scheduler that sends a request to whichever replica already holds its prefix, which turns cache locality into a load balancing problem. It is the same idea that shapes [inference optimization](https://howaiworks.ai/glossary/inference-optimization) generally: the cheapest work is work already done, and the hard part is knowing where it was done. ## Code Example No API keys and no vendor SDK — the mechanism is a cumulative hash and some arithmetic, and both are worth seeing directly. The first half builds the same logical prompt two ways and reports the block at which two consecutive requests stop sharing a prefix. The second half runs the break-even. ```python import hashlib import json def prefix_hashes(blocks): """A provider hashes the prompt cumulatively. Block i's hash covers blocks 0..i.""" running, out = hashlib.sha256(), [] for block in blocks: running.update(block.encode("utf-8")) out.append(running.hexdigest()[:8]) return out def first_divergence(request_a, request_b): """The block index where two requests stop sharing a prefix. Everything from here on is a cache miss, no matter how identical the later blocks look.""" for i, (a, b) in enumerate(zip(prefix_hashes(request_a), prefix_hashes(request_b))): if a != b: return i return None TOOLS = {"search": "query the docs", "run_sql": "read-only query", "email": "send mail"} SYSTEM = "You are a support agent for Acme. Answer only from the retrieved docs." def render(tools, system, messages, stamp=None): """Providers render tools, then system, then messages -- in that order.""" tool_block = json.dumps(tools, sort_keys=True) # deterministic serialisation system_block = system if stamp is None else f"{system}\nCurrent time: {stamp}" return [tool_block, system_block, *messages] # Cache-hostile: a timestamp sits in the system prompt, at position 1 of 3. hostile_1 = render(TOOLS, SYSTEM, ["Where is my order?"], stamp="2026-07-24T09:00:01Z") hostile_2 = render(TOOLS, SYSTEM, ["Where is my order?"], stamp="2026-07-24T09:00:02Z") # Cache-friendly: the same timestamp, moved into the volatile user turn. friendly_1 = render(TOOLS, SYSTEM, ["[2026-07-24T09:00:01Z] Where is my order?"]) friendly_2 = render(TOOLS, SYSTEM, ["[2026-07-24T09:00:02Z] Where is my order?"]) print("hostile -> first differing block:", first_divergence(hostile_1, hostile_2)) print("friendly -> first differing block:", first_divergence(friendly_1, friendly_2)) def cost_per_call(n_calls, prefix_tokens, base_per_mtok, write_mult, read_mult): """One cache write, then n-1 reads, versus paying base input every time.""" mtok = prefix_tokens / 1_000_000 uncached = n_calls * mtok * base_per_mtok cached = mtok * base_per_mtok * (write_mult + read_mult * (n_calls - 1)) return uncached, cached print() print(" calls uncached cached saving") for n in (1, 2, 3, 10, 100, 1000): uncached, cached = cost_per_call(n, 20_000, 5.00, 1.25, 0.10) print(f"{n:6d} ${uncached:8.3f} ${cached:8.3f} {100 * (1 - cached / uncached):7.1f}%") ``` Running it prints: ``` hostile -> first differing block: 1 friendly -> first differing block: 2 calls uncached cached saving 1 $ 0.100 $ 0.125 -25.0% 2 $ 0.200 $ 0.135 32.5% 3 $ 0.300 $ 0.145 51.7% 10 $ 1.000 $ 0.215 78.5% 100 $ 10.000 $ 1.115 88.8% 1000 $ 100.000 $ 10.115 89.9% ``` Both facts are visible in the output. The hostile version diverges at block **1** — the system prompt — so blocks 1 and 2 are both uncacheable, which in a real deployment means the entire system prompt and every turn of the conversation. The friendly version diverges at block **2**, the user message, so blocks 0 and 1 — tools and system, the large ones — cache cleanly. The only difference between the two is which string the timestamp was concatenated onto. The table shows why a single-shot prompt should never be marked for caching (a 25% loss) and why almost anything reused should be (32.5% by the second call, 89.9% by the thousandth, and never better than 90%). Swap `write_mult=2.00` for a longer-lived cache and the first two rows go negative, which is the TTL trade-off in two numbers rather than a paragraph of advice. ## Frequently Asked Questions ### What is prompt caching in simple terms? It is the provider keeping the processed form of the front of your prompt so the next request that starts with exactly the same bytes can skip re-reading it. You pay a small premium the first time to store it, and a fraction of the normal input price every time after that. ### Why is my cache hit rate zero? Almost always one of two things. Either the prefix is shorter than the provider's minimum cacheable length, in which case nothing caches and no error is raised, or something inside the prefix changes on every request — a timestamp, a UUID, a re-serialised tool list — so the byte match never succeeds. Check the response usage fields: if both the write and read counters are zero it is the minimum; if the write counter is non-zero every time and the read counter never is, it is a hidden invalidator. ### How many requests does it take for prompt caching to pay off? With a write premium of 1.25x base input and reads at 0.1x, break-even lands at 1.28 requests — the second request is already cheaper. With a longer-lived cache written at 2x, break-even moves to 2.11, so you need a third request. Either way the saving ceiling is 90%, because reads never get cheaper than the read multiplier. ### What is the difference between prompt caching and the KV cache? The KV cache is the within-request mechanism: keys and values a model stores as it reads, so it never recomputes the past. Prompt caching is the across-request product built on top of it — the provider keeps that state alive between your API calls and bills you for the reuse. Same physics, different billing surface. ### Where should the cache breakpoint go? On the last block that is byte-identical across the requests you want to share a cache, and never on a block that varies. Stable content — tool definitions, the frozen system prompt, few-shot examples, retrieved documents that change daily — goes above it; the incoming question, timestamps and per-request identifiers go below it. ### Does prompt caching make responses faster? It cuts time to first token, because the provider skips re-reading the cached prefix. It does nothing for the speed of generation itself — that phase reads model weights token by token and is unaffected by whether the prompt was cached. ## Related ### Related terms - [KV Cache](https://howaiworks.ai/glossary/kv-cache) - [Inference Optimization](https://howaiworks.ai/glossary/inference-optimization) - [Context Window](https://howaiworks.ai/glossary/context-window) - [Prompt Engineering](https://howaiworks.ai/glossary/prompt-engineering) - [Token](https://howaiworks.ai/glossary/token) --- Source: https://howaiworks.ai/glossary/prompt-caching — HowAIWorks.ai