Definition
Prompt caching bills the front of your prompt at a fraction of its normal price, provided the bytes have not changed by so much as one character. Large language model APIs are stateless: every request resends the whole conversation, and a coding agent on its fortieth turn is paying to re-read the same 30,000-token system prompt and tool list for the fortieth time. Prompt caching lets the provider keep the processed form of that repeated prefix, and charge you for storage-plus-reuse instead of re-reading. On Anthropic's published multipliers as of July 2026, a cache read costs 0.1x the base input price and a cache write 1.25x — so a 20,000-token prefix reused across a thousand calls drops from $100.00 to $10.12, a 89.9% saving.
The catch, and the reason most teams who switch it on see nothing, is the qualifier. Caching is a
prefix match: the provider hashes your prompt from byte zero, and a single differing character
at position N destroys the match for everything after N. Interpolate datetime.now() into the top
of your system prompt and you have not merely lost the discount — you now pay the 1.25x write
premium on every single call and never once read, which is 25% more than never caching at all.
Nothing errors. The bill just goes up.
How It Works
Everything follows from the prefix match
There is exactly one rule on this page, and every practical consequence is derived from it. The cache key is a cumulative hash of the rendered prompt up to the point you mark. Block five's hash covers blocks one through five. So the question a provider asks is never "have I seen this system prompt before?" — it is "have I seen this exact sequence of bytes from the beginning before?"
That reframing explains behaviour that otherwise looks arbitrary:
A timestamp anywhere in the prefix invalidates everything behind it. Not just the line it sits on. If the current date is rendered into the system prompt at position 1 of 3, then blocks 2 and 3 are unreachable, even though they are byte-identical to last time. The 20,000 tokens of frozen instructions sitting after the timestamp are re-read at full price on every request, forever.
Non-deterministic serialisation does the same thing, invisibly. json.dumps() on a Python dict
emits keys in insertion order; build the dict by a different code path and the same data serialises
to different bytes. Iterating a set is worse — CPython randomises string hashing per process, so
your tool list can serialise differently after a restart. The prompt is semantically identical, the
hash is not, and nothing in your logs will say so.
Changing the tool list invalidates the entire conversation. Providers render the prompt in a
fixed order — Anthropic documents it as tools, then system, then messages — which puts tool
definitions at position zero. Adding one tool to an agent mid-session does not
invalidate "the tools section". It invalidates the tools, the system prompt, and every turn of the
conversation built on top of them. This is why "modes" that swap the
tool set per request are a caching disaster, and why building tools
per-user means no two users ever share a cache entry.
So stable content must physically precede volatile content. That is the whole design rule. Frozen system prompt and deterministic tool list first, then slowly-changing context (retrieved documents, few-shot examples), then the per-request material — the user's question, the timestamp, the session ID — last. Get the ordering right and caching mostly works for free; get it wrong and no amount of configuration will rescue it.
The breakpoint, and why writes only happen where you mark
Marking a block tells the provider to write one cache entry: the hash of the prefix ending at that block. It does not write entries at earlier positions. On the next request the provider computes the hash at your mark, and if it misses, walks backward looking for an entry an earlier request wrote — Anthropic caps that walk at 20 blocks.
The consequence trips people up. If you mark the last block of a request whose last block is the user's question, you write a distinct entry every time and read none of them, because the lookback finds no prior write behind it. It is not searching for stable content; it is searching for previous writes. The mark belongs on the last stable block, not the last block.
The 20-block limit has its own sharp edge in long agent runs. If a single turn appends more than 20 blocks — easy when a turn fires ten tool calls and receives ten results — the next request's lookback never reaches the previous turn's entry, and a conversation that was hitting cache all morning quietly stops. Anthropic allows up to 4 breakpoints per request precisely so you can plant an intermediate mark inside long turns.
Deriving the break-even
Take one unit as the cost of processing the prefix at the base input price. Over N requests:
uncached = N
cached = w + r x (N - 1) w = write multiplier, r = read multiplier
They are equal when N = (w - r) / (1 - r). With Anthropic's w = 1.25 and r = 0.10, that is
1.28 requests — meaning the second request is already cheaper, and the first one alone costs you
25% extra. Choose the longer-lived cache, written at w = 2, and break-even moves to 2.11: the
second request is a small loss and the third pays. That is the real trade in TTL choice. A
longer-lived entry survives gaps in bursty traffic, but the doubled write means it needs more reads
to earn its keep.
The ceiling is worth naming too. As N grows, cached / uncached approaches r, so the saving
asymptotes at 90% and never exceeds it. Prompt caching does not make the prefix free; it makes
it a tenth.
Where a template edit actually costs you
Both MLOps and monitoring practice warn that a prompt template edit adding 400 tokens raises the bill on every call, forever, and breaks no test. Caching changes that warning in a way that is worth internalising: with a cache in play, where the tokens go matters roughly ten times more than how many there are.
Take that 20,000-token prefix on an Opus-tier model at $5 per million input tokens (July 2026), reused often enough to sit at steady state. The cached read costs $0.0100 per call. Now add the same 400 tokens three different ways:
- Static, above the breakpoint. The prefix becomes 20,400 tokens and still caches. Per call: $0.0102 — a 2% rise.
- Static, below the breakpoint. Those tokens never cache, so they bill at the full base rate on every call. Per call: $0.0120 — a 20% rise, ten times worse than the identical bytes placed above the mark.
- Volatile, above the breakpoint (a rendered timestamp, a request ID). Nothing matches, so every call is a fresh 20,400-token write at 1.25x. Per call: $0.1275 — 12.75x the baseline.
Same 400 tokens; +2%, +20%, or +1,175% depending only on position. The reviewable artefact is no longer the diff's size, it is where in the prompt the diff landed.
It is not the same thing as the KV cache
These two get conflated constantly, and the distinction is clean. The KV cache
is the within-request mechanism: as the model reads, each layer stores a key and a value vector
per token so later tokens can attend to the past without recomputing it. It is why generating the
second word of an answer is cheap. Prompt caching is the across-request product built on that
mechanism: the provider keeps those keys and values resident between your API calls and sells you
the right to skip the read-in phase. Same physics, different billing surface — and only one of them
is something you can break with a datetime.now().
Real-World Applications
Coding agents and long-running assistants. This is where caching stopped being an optimisation and became a precondition. An agentic workflow that reads a repository and then takes forty turns resends the entire accumulated transcript on every turn; at turn forty the prompt is mostly things the model has already been shown thirty-nine times. Without caching the cost of a session grows with the square of its length, since each of N turns resends O(N) of history. With caching, the repeated part bills at a tenth and the growth curve flattens to something a product can be priced on. It is not a coincidence that per-turn pricing for coding assistants arrived at the same time as prompt caching did.
Retrieval over a corpus that changes slower than the questions. RAG systems answering many questions against the same policy manual, contract set or knowledge base can put the retrieved documents above the breakpoint and the question below it. The ordering is the whole design decision, and it is the opposite of what a naive implementation does — most templates put retrieved chunks last, right before the answer, because that reads better. Reading better costs 10x.
Classification and extraction at volume. A pipeline that runs the same 8,000-token instruction block plus twenty few-shot examples across a million documents is the ideal shape: an enormous fixed prefix, a tiny variable suffix. Break-even arrives on the second document and the remaining 999,998 run at a tenth of the input price.
Deciding whether a longer prompt is affordable. Caching changes the answer to "can we afford to put the whole style guide in the system prompt?" from no to usually, provided the guide is static. That is a genuine capability shift, not just a discount — the constraint on system-prompt size moves from cost toward the context window itself.
Key Concepts
Cache write, cache read, and uncached input are three different line items with three different
prices, and every provider that supports caching reports them as separate counters in the response.
Anthropic's are cache_creation_input_tokens, cache_read_input_tokens and input_tokens, and the
relationship worth memorising is that they sum to the total prompt:
total input = cache_read + cache_creation + input_tokens
input_tokens is only the remainder after your last breakpoint, which is why an agent that ran all
morning can show a suspiciously small input_tokens — the rest was served from cache. Reading that
field alone as "how big was my prompt" is a reliable way to misjudge both cost and rate limits.
TTL is how long an unused entry survives. Anthropic's default is five minutes, refreshed at no charge every time the entry is read, with a longer one-hour option at the doubled write price. The refresh-on-read behaviour matters more than the raw number: traffic arriving more often than the TTL keeps the cache alive indefinitely at no extra cost, so the longer TTL only earns its premium for genuinely bursty workloads with idle gaps.
Minimum cacheable length is a floor below which nothing caches at all. Anthropic's published minimums as of July 2026 range from 512 to 4,096 tokens depending on the model; Google documents a per-model minimum for Gemini's implicit context caching too. It is not a quirk of one vendor — it exists because bookkeeping for a tiny entry costs more than the entry saves.
Challenges
The silent failure: the cache never populates. Your prefix is below the minimum cacheable length for the model you are calling. There is no error, no warning, and no field that says "too short" — the request is simply processed without caching. The tell is that both the write and read counters come back zero, request after request. This bites hardest when a prompt sits just under the threshold, and the fix is counter-intuitive: make the cached section longer until it clears the floor, since a read is so much cheaper than uncached input that padding up to the minimum can lower the bill. It also bites when you change models — the minimum is per-model, so a prefix that cached happily on one model can silently stop on another, with nothing in the diff to blame.
The invisible failure: you write forever and never read. Something in the prefix varies per
request, so every call writes a fresh entry and no call ever matches one. This is strictly worse
than not caching: you pay 1.25x base input on every request instead of 1.00x, a permanent 25%
surcharge, while your configuration says caching is enabled and your dashboards look fine. The
diagnostic is one line — if cache_creation_input_tokens is non-zero on every request and
cache_read_input_tokens is never non-zero, hunt the invalidator. Candidates, in rough order of
frequency: a rendered timestamp or date, a UUID or request ID, a user or session identifier
interpolated into the system prompt, unsorted JSON, a conditionally-assembled system prompt where
each feature-flag combination is a distinct prefix, and a tool list built per-user.
Forks and sub-agents rebuild the prefix and lose it. A summarisation pass, a compaction step or a spawned sub-agent typically constructs its own request. If it rebuilds the system prompt or the tool list with any difference — a different assembly order, a trimmed instruction, a different model — it misses the parent's cache entirely and pays a full cold write. The fix is unglamorous: copy the parent's prefix verbatim and append the fork-specific material at the end.
Parallel requests all miss. An entry only becomes readable once the first response begins, so firing N identical-prefix requests simultaneously means N cache writes and zero reads — the expensive outcome, arrived at by doing the obviously efficient thing. Fan-out patterns need to send one request, wait for it to start returning, then release the rest.
Cheap tokens are still tokens. Cached input costs a tenth but occupies the context window in full, counts against rate limits, and does nothing for generation speed — caching cuts time to first token, not the per-token decode that dominates a long answer. Teams that discover the discount sometimes respond by stuffing the prefix with everything that might conceivably help, and rediscover that a bloated context degrades answer quality at a tenth of the price.
Future Trends
Automatic caching is absorbing the breakpoint. Providers increasingly place the mark for you — Google's Gemini caching is implicit by default, and Anthropic offers a top-level automatic mode that moves the breakpoint forward as a conversation grows. This removes the most common mistake and introduces a subtler one: automatic placement puts the mark on the last cacheable block, which in a prompt with a varying suffix is precisely the block that changes every request. The failure mode migrates from "I forgot to mark anything" to "it marked the wrong thing and I did not notice."
Prompt layout is becoming a reviewed artefact. Once the difference between a 2% and a 1,175% cost change is where a string was inserted, prompt assembly stops being template code and starts being something with an invariant worth testing — a unit test that renders the prefix twice and asserts the bytes match is trivial to write and catches every invalidator on the list above. Expect that assertion to become as routine in LLM services as a schema check.
The API surface is growing affordances specifically to protect the prefix. The clearest example is mid-conversation system messages: instead of editing the top-level system prompt when an operator instruction arrives mid-session — which changes byte zero and re-processes the entire conversation — providers now let you append a system-role message after the cached history. Watch for more of this shape: features whose entire purpose is to let something change without changing the front of the prompt.
Cache-aware routing. In self-hosted serving, the natural next step is a scheduler that sends a request to whichever replica already holds its prefix, which turns cache locality into a load balancing problem. It is the same idea that shapes inference optimization generally: the cheapest work is work already done, and the hard part is knowing where it was done.
Code Example
No API keys and no vendor SDK — the mechanism is a cumulative hash and some arithmetic, and both are worth seeing directly. The first half builds the same logical prompt two ways and reports the block at which two consecutive requests stop sharing a prefix. The second half runs the break-even.
import hashlib
import json
def prefix_hashes(blocks):
"""A provider hashes the prompt cumulatively. Block i's hash covers blocks 0..i."""
running, out = hashlib.sha256(), []
for block in blocks:
running.update(block.encode("utf-8"))
out.append(running.hexdigest()[:8])
return out
def first_divergence(request_a, request_b):
"""The block index where two requests stop sharing a prefix. Everything from
here on is a cache miss, no matter how identical the later blocks look."""
for i, (a, b) in enumerate(zip(prefix_hashes(request_a), prefix_hashes(request_b))):
if a != b:
return i
return None
TOOLS = {"search": "query the docs", "run_sql": "read-only query", "email": "send mail"}
SYSTEM = "You are a support agent for Acme. Answer only from the retrieved docs."
def render(tools, system, messages, stamp=None):
"""Providers render tools, then system, then messages -- in that order."""
tool_block = json.dumps(tools, sort_keys=True) # deterministic serialisation
system_block = system if stamp is None else f"{system}\nCurrent time: {stamp}"
return [tool_block, system_block, *messages]
# Cache-hostile: a timestamp sits in the system prompt, at position 1 of 3.
hostile_1 = render(TOOLS, SYSTEM, ["Where is my order?"], stamp="2026-07-24T09:00:01Z")
hostile_2 = render(TOOLS, SYSTEM, ["Where is my order?"], stamp="2026-07-24T09:00:02Z")
# Cache-friendly: the same timestamp, moved into the volatile user turn.
friendly_1 = render(TOOLS, SYSTEM, ["[2026-07-24T09:00:01Z] Where is my order?"])
friendly_2 = render(TOOLS, SYSTEM, ["[2026-07-24T09:00:02Z] Where is my order?"])
print("hostile -> first differing block:", first_divergence(hostile_1, hostile_2))
print("friendly -> first differing block:", first_divergence(friendly_1, friendly_2))
def cost_per_call(n_calls, prefix_tokens, base_per_mtok, write_mult, read_mult):
"""One cache write, then n-1 reads, versus paying base input every time."""
mtok = prefix_tokens / 1_000_000
uncached = n_calls * mtok * base_per_mtok
cached = mtok * base_per_mtok * (write_mult + read_mult * (n_calls - 1))
return uncached, cached
print()
print(" calls uncached cached saving")
for n in (1, 2, 3, 10, 100, 1000):
uncached, cached = cost_per_call(n, 20_000, 5.00, 1.25, 0.10)
print(f"{n:6d} ${uncached:8.3f} ${cached:8.3f} {100 * (1 - cached / uncached):7.1f}%")
Running it prints:
hostile -> first differing block: 1
friendly -> first differing block: 2
calls uncached cached saving
1 $ 0.100 $ 0.125 -25.0%
2 $ 0.200 $ 0.135 32.5%
3 $ 0.300 $ 0.145 51.7%
10 $ 1.000 $ 0.215 78.5%
100 $ 10.000 $ 1.115 88.8%
1000 $ 100.000 $ 10.115 89.9%
Both facts are visible in the output. The hostile version diverges at block 1 — the system prompt — so blocks 1 and 2 are both uncacheable, which in a real deployment means the entire system prompt and every turn of the conversation. The friendly version diverges at block 2, the user message, so blocks 0 and 1 — tools and system, the large ones — cache cleanly. The only difference between the two is which string the timestamp was concatenated onto.
The table shows why a single-shot prompt should never be marked for caching (a 25% loss) and why
almost anything reused should be (32.5% by the second call, 89.9% by the thousandth, and never
better than 90%). Swap write_mult=2.00 for a longer-lived cache and the first two rows go
negative, which is the TTL trade-off in two numbers rather than a paragraph of advice.