--- source: 'https://howaiworks.ai/glossary/kv-cache' section: glossary title: KV Cache description: >- The memory an LLM keeps so it never recomputes the past — why a long context costs money, and how many users one GPU can actually serve at once. tags: - Inference - LLM - GPU - Memory Bandwidth - Long Context - Inference Optimization - AI Infrastructure - transformer category: Infrastructure & Hardware datePublished: '2026-07-13' lastUpdated: '2026-07-24' --- # KV Cache > The memory an LLM keeps so it never recomputes the past — why a long context costs money, and how many users one GPU can actually serve at once. ## Definition **The KV cache is the reason the second token of an answer is cheap and the 128,000-token prompt in front of it is expensive.** As a language model reads text, every layer turns each [token](https://howaiworks.ai/glossary/token) into a *key* and a *value* vector — the things later tokens look at when they attend to the past. The model saves them. That saved pile is the KV cache, and it is why generating the next word costs one token's worth of work instead of re-reading the entire conversation from the beginning. The catch is its size. For Meta's Llama 3.1 70B, the cache costs **320 KiB per token**, so one user at the model's full 128K context is holding about **43 GB of GPU memory** — half again more than most people expect, and enough that an eight-GPU server which fits the model's weights with room to spare still tops out at around **eleven** such sessions. A million-token context is a *memory* problem long before it is an attention problem, and that is the whole story of this page. ## How It Works ### Without the cache, generation is quadratic [Self-attention](https://howaiworks.ai/glossary/self-attention) lets each token look at every token before it, which means the model needs the keys and values of the entire history at every step. It could recompute them. Watch what that costs. Take a 2,000-token prompt and a 1,000-token answer. **With** a cache, the model runs one forward pass over the 2,000 prompt tokens, then 1,000 passes over one token each: about **3,000 token-forwards**. **Without** one, every new token forces a fresh pass over everything written so far — 2,000 tokens for the first, 2,001 for the second, up to 2,999 for the last. That sum is **about 2.5 million token-forwards**, roughly **800x** the work, and the gap widens with the square of the output length: double the answer and the recompute quadruples. So the cache is not an optimisation anyone chose to bolt on. It is the thing that makes autoregressive [inference](https://howaiworks.ai/glossary/inference) tractable at all. What it does is trade compute for memory — and memory is the resource that turns out to be scarce. ### Prefill and decode are two different machines That trade splits serving into two phases with almost nothing in common. **Prefill** reads the prompt. Every token is available at once, so the model processes them in parallel — tall, dense [matrix multiplications](https://howaiworks.ai/glossary/matrix-multiplication) that keep a GPU's arithmetic units genuinely busy — and writes the resulting keys and values into the cache. It is compute-bound, it scales with prompt length, and it produces exactly one token of output. **Decode** writes the answer. It generates one token, appends its key and value to the cache, and starts again. To produce that single token the GPU must read *all* of the model's weights and *all* of the cache built so far, then do a trivial amount of arithmetic with them. The bottleneck is not maths, it is fetching bytes from memory — which is why decode is memory-bound, why batching many users together is the only way to make it efficient, and why a GPU can look busy while its arithmetic units idle. The reason that distinction dominates modern inference economics belongs to the [memory wall](https://howaiworks.ai/glossary/memory-wall); here it is enough to know that prefill and decode pull in opposite directions, and that the largest serving stacks now run them on *separate machines* for exactly that reason. ### The size formula, and the number it produces For a standard transformer, the cache is: ``` bytes per token = 2 (K and V) x layers x KV heads x head dimension x bytes per number ``` Note what is *not* in it: the number of query heads, the hidden size, the FFN width. The cache is sized by the KV heads alone — which is the hinge the whole industry pushes on. Now run it on a real model. Meta's published config for **Llama 3.1 70B** gives 80 layers, 64 attention heads, **8 key/value heads**, hidden size 8192, and a 131,072-token context, trained and served in bfloat16. Head dimension is 8192 / 64 = **128**. So: ``` 2 x 80 layers x 8 KV heads x 128 dims x 2 bytes = 327,680 bytes = 320 KiB per token ``` - **One user at full context**: 327,680 bytes x 131,072 tokens = **42.9 GB** (exactly 40 GiB). - **The model's weights**: about 70.6B parameters x 2 bytes = **141 GB**. - **The server**: eight H100 SXM GPUs, 80 GB each, is **640 GB** of HBM. - **What is left for cache**: 640 − 141 = **499 GB**. - **Concurrent full-context sessions**: 499 / 42.9 = **11.6, call it 11.** Eleven. On a machine that costs as much as a house, running a model whose weights use barely a fifth of its memory. That is the number to keep: **the weights are a fixed cost paid once, the cache is a variable cost paid per user per token**, and past a certain context length the variable cost is the entire budget. ### Fewer KV heads: multi-query and grouped-query attention Because the formula multiplies by KV heads, cutting them cuts the cache proportionally — and the same eight-GPU box changes character completely: - **Multi-head attention** (one KV head per query head — 64 of them). Noam Shazeer's 2019 paper named this as the bottleneck for exactly this reason. Cache: 8x bigger, **343 GB per session**. That is *more than twice the model's weights*, for one user. The box serves **one** person. - **Multi-query attention** (a single KV head shared by all 64 query heads, Shazeer's fix). Cache: **5.4 GB per session**. The box serves **92**. But sharing one head across all queries costs accuracy. - **Grouped-query attention** — what Llama 3.1 actually uses: 8 KV heads, each shared by 8 query heads. The 2023 GQA paper describes it as reaching "quality close to multi-head attention with comparable speed to MQA." Cache: **43 GB**, and eleven users. One architectural choice, an 8x swing in serving capacity, and no change to the parameter count. This is why every frontier model shipped since 2023 uses GQA or something stronger — DeepSeek-V2's multi-head latent attention compresses keys and values into a shared low-rank vector and reports a **93.3% cache reduction** against its own predecessor. Storing the cache in fewer bits is the other axis, and it belongs to [quantization](https://howaiworks.ai/glossary/quantization). ### Paged attention: stop wasting the remainder Even with the cache small, serving systems used to allocate it badly. A request that *might* run to 128K tokens was given a contiguous 128K-token block up front, and a reply that ended after 300 tokens left the rest stranded — reserved, unused, unusable by anyone else. The vLLM team measured the damage: existing systems **wasted 60–80% of KV cache memory** to fragmentation and over-reservation. **PagedAttention** fixes it with an idea borrowed wholesale from operating systems: chop the cache into fixed-size blocks, keep a page table mapping a sequence's logical positions to physical blocks scattered anywhere in memory, and allocate a block only when it is actually needed. Waste falls to **under 4%**, and because the memory you recover immediately becomes larger batches, throughput rises **2–4x at the same latency**. It is now the default in essentially every open-source serving stack. ## Real-World Applications **The price of a cached prompt.** ([Prompt caching](https://howaiworks.ai/glossary/prompt-caching) is the product built on this mechanism.) When Anthropic's API charges **0.1x** the normal input price for cached prefix tokens and **1.25x** to write the cache, that pricing is not a discount scheme — it is the KV cache, kept resident between requests. A cache read is the provider skipping prefill because the keys and values for your system prompt are still sitting in HBM; the write premium is the cost of putting them there. A 90% saving on the repeated part of every agent turn is a direct consequence of the mechanism on this page, which is why prompt-caching-aware prompt layout (stable content first, variable content last) is worth real money. **Sizing a deployment.** "Will this model fit on this GPU?" is the wrong question, and teams ask it constantly. A 70B model in bfloat16 fits on two H100s, weights-wise. Two H100s serve roughly *zero* users at 128K context, because 160 GB of memory minus 141 GB of weights leaves 19 GB and one session needs 43. The real question is *weights plus cache times concurrency*, and getting it wrong is the most common way an inference deployment fails in its first week. **Why context windows are priced and capped the way they are.** Every provider's maximum context, maximum concurrency, and long-context surcharge are downstream of this arithmetic. When a lab ships a 1M-token [context window](https://howaiworks.ai/glossary/context-window), the engineering underneath it is overwhelmingly cache engineering — compression, sparsity, offloading to CPU memory — not a new [attention mechanism](https://howaiworks.ai/glossary/attention-mechanism). ## Challenges **You size the GPU by the weights and it dies under load.** This is the failure mode. The model loads, a single test request works beautifully, and the service falls over at eight concurrent users because nobody multiplied 43 GB by eight. There is no warning: memory pressure shows up as requests being queued or evicted, not as an error you can read. **The cache is the reason your throughput collapses at long context, not the attention maths.** Teams profile, find attention is fine, and miss that the batch size their server can sustain has quietly dropped from 64 to 3 because each sequence now carries tens of gigabytes. Latency per token barely moves; tokens per second per *dollar* falls off a cliff. **Eviction has no good answer.** When memory runs out, a serving system must either preempt a running request (recomputing its cache later, paying the prefill twice) or swap the cache to host memory across PCIe, which is roughly an order of magnitude slower than HBM. Both are bad; the choice is which kind of bad. **Quality-for-memory trades are real trades.** Fewer KV heads, fewer bits, and evicting "unimportant" tokens all shrink the cache and all cost something — usually invisibly, on the long-context retrieval tasks the compression was supposed to enable. ## Future Trends The clearest structural shift is **disaggregated serving**: run prefill on machines chosen for compute and decode on machines chosen for memory bandwidth, then ship the KV cache between them over the network. It is an admission that the two phases were never the same workload, and it turns the cache into something that gets *transferred*, which in turn makes cache placement a scheduling problem. The second is treating the cache as a **storage tier rather than a buffer**. Prefix caches shared across users and across requests, spilled to CPU RAM or NVMe and pulled back when a conversation resumes, are already how large agent deployments avoid re-prefilling the same 50,000-token codebase a hundred times an hour. The interesting consequence is architectural: once the cache is a durable, addressable, deduplicated object, the boundary between "context" and "memory" — in the human sense — starts to blur, and a model's working set stops being something it rebuilds from scratch every time you say hello. ## Frequently Asked Questions ### What is a KV cache in simple terms? It is the model's notes on everything it has already read. Each token a transformer processes produces a key and a value vector in every layer; storing them means the next token can attend to the whole history without the model re-reading it. Drop the cache and the model has to recompute the entire conversation for every single word it writes. ### How big is the KV cache? Bytes per token = 2 x layers x KV heads x head dimension x bytes per number, then multiply by the number of tokens and by the number of concurrent users. For Llama 3.1 70B in bfloat16 that is 320 KiB per token, so a single session at its full 128K context holds about 43 GB — more than half an H100's memory, for one user. ### Why does a long context cost more per request? Two reasons, and only one of them is compute. Reading the prompt is a one-off cost, but the resulting cache has to be held in GPU memory for the whole generation, and it grows linearly with length. Memory held is memory nobody else can use, so a long-context request occupies a slot that could have served several short ones. ### What is the difference between prefill and decode? Prefill processes every prompt token at once and fills the cache; it is a big parallel matmul and it is compute-bound. Decode then produces one token at a time, reading the model's weights and the whole cache for each one; it is memory-bound. They have such different cost profiles that serving stacks increasingly run them on separate machines. ### Does prompt caching use the KV cache? Yes — it is the same mechanism, kept alive between requests. When an API charges a fraction of the normal input price for a cached prefix, what you are buying is the right to skip prefill because the keys and values for that prefix are still resident in GPU memory. ### How do you make the KV cache smaller? Store fewer KV heads (multi-query and grouped-query attention), store them in fewer bits, compress them into a latent vector (as DeepSeek's multi-head latent attention does), or stop wasting the memory you allocated but never filled (paged attention). Most production models use at least two of these at once. ## Related ### Related terms - [Context Window](https://howaiworks.ai/glossary/context-window) - [Attention Mechanism](https://howaiworks.ai/glossary/attention-mechanism) - [Inference](https://howaiworks.ai/glossary/inference) - [Transformer](https://howaiworks.ai/glossary/transformer) - [Token](https://howaiworks.ai/glossary/token) - [Self-Attention](https://howaiworks.ai/glossary/self-attention) --- Source: https://howaiworks.ai/glossary/kv-cache — HowAIWorks.ai