Definition
The KV cache is the reason the second token of an answer is cheap and the 128,000-token prompt in front of it is expensive. As a language model reads text, every layer turns each token into a key and a value vector — the things later tokens look at when they attend to the past. The model saves them. That saved pile is the KV cache, and it is why generating the next word costs one token's worth of work instead of re-reading the entire conversation from the beginning.
The catch is its size. For Meta's Llama 3.1 70B, the cache costs 320 KiB per token, so one user at the model's full 128K context is holding about 43 GB of GPU memory — half again more than most people expect, and enough that an eight-GPU server which fits the model's weights with room to spare still tops out at around eleven such sessions. A million-token context is a memory problem long before it is an attention problem, and that is the whole story of this page.
How It Works
Without the cache, generation is quadratic
Self-attention lets each token look at every token before it, which means the model needs the keys and values of the entire history at every step. It could recompute them. Watch what that costs.
Take a 2,000-token prompt and a 1,000-token answer. With a cache, the model runs one forward pass over the 2,000 prompt tokens, then 1,000 passes over one token each: about 3,000 token-forwards. Without one, every new token forces a fresh pass over everything written so far — 2,000 tokens for the first, 2,001 for the second, up to 2,999 for the last. That sum is about 2.5 million token-forwards, roughly 800x the work, and the gap widens with the square of the output length: double the answer and the recompute quadruples.
So the cache is not an optimisation anyone chose to bolt on. It is the thing that makes autoregressive inference tractable at all. What it does is trade compute for memory — and memory is the resource that turns out to be scarce.
Prefill and decode are two different machines
That trade splits serving into two phases with almost nothing in common.
Prefill reads the prompt. Every token is available at once, so the model processes them in parallel — tall, dense matrix multiplications that keep a GPU's arithmetic units genuinely busy — and writes the resulting keys and values into the cache. It is compute-bound, it scales with prompt length, and it produces exactly one token of output.
Decode writes the answer. It generates one token, appends its key and value to the cache, and starts again. To produce that single token the GPU must read all of the model's weights and all of the cache built so far, then do a trivial amount of arithmetic with them. The bottleneck is not maths, it is fetching bytes from memory — which is why decode is memory-bound, why batching many users together is the only way to make it efficient, and why a GPU can look busy while its arithmetic units idle. The reason that distinction dominates modern inference economics belongs to the memory wall; here it is enough to know that prefill and decode pull in opposite directions, and that the largest serving stacks now run them on separate machines for exactly that reason.
The size formula, and the number it produces
For a standard transformer, the cache is:
bytes per token = 2 (K and V) x layers x KV heads x head dimension x bytes per number
Note what is not in it: the number of query heads, the hidden size, the FFN width. The cache is sized by the KV heads alone — which is the hinge the whole industry pushes on.
Now run it on a real model. Meta's published config for Llama 3.1 70B gives 80 layers, 64 attention heads, 8 key/value heads, hidden size 8192, and a 131,072-token context, trained and served in bfloat16. Head dimension is 8192 / 64 = 128. So:
2 x 80 layers x 8 KV heads x 128 dims x 2 bytes = 327,680 bytes = 320 KiB per token
- One user at full context: 327,680 bytes x 131,072 tokens = 42.9 GB (exactly 40 GiB).
- The model's weights: about 70.6B parameters x 2 bytes = 141 GB.
- The server: eight H100 SXM GPUs, 80 GB each, is 640 GB of HBM.
- What is left for cache: 640 − 141 = 499 GB.
- Concurrent full-context sessions: 499 / 42.9 = 11.6, call it 11.
Eleven. On a machine that costs as much as a house, running a model whose weights use barely a fifth of its memory. That is the number to keep: the weights are a fixed cost paid once, the cache is a variable cost paid per user per token, and past a certain context length the variable cost is the entire budget.
Fewer KV heads: multi-query and grouped-query attention
Because the formula multiplies by KV heads, cutting them cuts the cache proportionally — and the same eight-GPU box changes character completely:
- Multi-head attention (one KV head per query head — 64 of them). Noam Shazeer's 2019 paper named this as the bottleneck for exactly this reason. Cache: 8x bigger, 343 GB per session. That is more than twice the model's weights, for one user. The box serves one person.
- Multi-query attention (a single KV head shared by all 64 query heads, Shazeer's fix). Cache: 5.4 GB per session. The box serves 92. But sharing one head across all queries costs accuracy.
- Grouped-query attention — what Llama 3.1 actually uses: 8 KV heads, each shared by 8 query heads. The 2023 GQA paper describes it as reaching "quality close to multi-head attention with comparable speed to MQA." Cache: 43 GB, and eleven users.
One architectural choice, an 8x swing in serving capacity, and no change to the parameter count. This is why every frontier model shipped since 2023 uses GQA or something stronger — DeepSeek-V2's multi-head latent attention compresses keys and values into a shared low-rank vector and reports a 93.3% cache reduction against its own predecessor. Storing the cache in fewer bits is the other axis, and it belongs to quantization.
Paged attention: stop wasting the remainder
Even with the cache small, serving systems used to allocate it badly. A request that might run to 128K tokens was given a contiguous 128K-token block up front, and a reply that ended after 300 tokens left the rest stranded — reserved, unused, unusable by anyone else. The vLLM team measured the damage: existing systems wasted 60–80% of KV cache memory to fragmentation and over-reservation.
PagedAttention fixes it with an idea borrowed wholesale from operating systems: chop the cache into fixed-size blocks, keep a page table mapping a sequence's logical positions to physical blocks scattered anywhere in memory, and allocate a block only when it is actually needed. Waste falls to under 4%, and because the memory you recover immediately becomes larger batches, throughput rises 2–4x at the same latency. It is now the default in essentially every open-source serving stack.
Real-World Applications
The price of a cached prompt. When Anthropic's API charges 0.1x the normal input price for cached prefix tokens and 1.25x to write the cache, that pricing is not a discount scheme — it is the KV cache, kept resident between requests. A cache read is the provider skipping prefill because the keys and values for your system prompt are still sitting in HBM; the write premium is the cost of putting them there. A 90% saving on the repeated part of every agent turn is a direct consequence of the mechanism on this page, which is why prompt-caching-aware prompt layout (stable content first, variable content last) is worth real money.
Sizing a deployment. "Will this model fit on this GPU?" is the wrong question, and teams ask it constantly. A 70B model in bfloat16 fits on two H100s, weights-wise. Two H100s serve roughly zero users at 128K context, because 160 GB of memory minus 141 GB of weights leaves 19 GB and one session needs 43. The real question is weights plus cache times concurrency, and getting it wrong is the most common way an inference deployment fails in its first week.
Why context windows are priced and capped the way they are. Every provider's maximum context, maximum concurrency, and long-context surcharge are downstream of this arithmetic. When a lab ships a 1M-token context window, the engineering underneath it is overwhelmingly cache engineering — compression, sparsity, offloading to CPU memory — not a new attention mechanism.
Challenges
You size the GPU by the weights and it dies under load. This is the failure mode. The model loads, a single test request works beautifully, and the service falls over at eight concurrent users because nobody multiplied 43 GB by eight. There is no warning: memory pressure shows up as requests being queued or evicted, not as an error you can read.
The cache is the reason your throughput collapses at long context, not the attention maths. Teams profile, find attention is fine, and miss that the batch size their server can sustain has quietly dropped from 64 to 3 because each sequence now carries tens of gigabytes. Latency per token barely moves; tokens per second per dollar falls off a cliff.
Eviction has no good answer. When memory runs out, a serving system must either preempt a running request (recomputing its cache later, paying the prefill twice) or swap the cache to host memory across PCIe, which is roughly an order of magnitude slower than HBM. Both are bad; the choice is which kind of bad.
Quality-for-memory trades are real trades. Fewer KV heads, fewer bits, and evicting "unimportant" tokens all shrink the cache and all cost something — usually invisibly, on the long-context retrieval tasks the compression was supposed to enable.
Future Trends
The clearest structural shift is disaggregated serving: run prefill on machines chosen for compute and decode on machines chosen for memory bandwidth, then ship the KV cache between them over the network. It is an admission that the two phases were never the same workload, and it turns the cache into something that gets transferred, which in turn makes cache placement a scheduling problem.
The second is treating the cache as a storage tier rather than a buffer. Prefix caches shared across users and across requests, spilled to CPU RAM or NVMe and pulled back when a conversation resumes, are already how large agent deployments avoid re-prefilling the same 50,000-token codebase a hundred times an hour. The interesting consequence is architectural: once the cache is a durable, addressable, deduplicated object, the boundary between "context" and "memory" — in the human sense — starts to blur, and a model's working set stops being something it rebuilds from scratch every time you say hello.