--- source: 'https://howaiworks.ai/glossary/grouped-query-attention' section: glossary title: Grouped-Query Attention (GQA) description: >- Grouped-query attention lets query heads share key/value heads, shrinking the KV cache that dominates long-context inference at near multi-head quality. tags: - attention mechanism - Inference - LLM - Long Context - Inference Optimization - transformer - Memory Bandwidth - AI Infrastructure category: Infrastructure & Hardware datePublished: '2026-07-25' lastUpdated: '2026-07-25' --- # Grouped-Query Attention (GQA) > Grouped-query attention lets query heads share key/value heads, shrinking the KV cache that dominates long-context inference at near multi-head quality. ## Definition **Grouped-query attention (GQA) is the reason a 70B model can serve roughly eight times as many users on the same GPU as it could with textbook attention, while answering almost as well.** In a standard transformer, every one of the model's attention heads keeps its own running notes on the conversation — a *key* and a *value* vector per [token](https://howaiworks.ai/glossary/token), per layer, held in memory for as long as the session lives. That pile of notes is the [KV cache](https://howaiworks.ai/glossary/kv-cache), and at long context it, not the model's weights, is what fills up the GPU. GQA shrinks it by a simple structural change: instead of every query head owning a private key/value head, query heads are sorted into *groups*, and each group shares one key/value head. The number that follows from that change is exact and durable: the cache shrinks by the ratio of query heads to key/value heads. Cut 64 key/value heads down to 8 and the cache is eight times smaller — for one architectural decision that leaves the parameter count essentially unchanged and, in careful measurements, barely moves output quality. That trade is why GQA is not an exotic optimisation but the default in nearly every open model shipped since 2023. ## How It Works ### The thing GQA is attacking To generate each new word, a transformer's [self-attention](https://howaiworks.ai/glossary/self-attention) lets the current position look back at every earlier one. To avoid re-reading the whole history at every step, the model stores the key and value vectors it already computed. The size of that store, for a standard model, is: ``` bytes per token = 2 (keys and values) x layers x KV heads x head dimension x bytes per number ``` The quantity to stare at is **KV heads**. The number of *query* heads is not in the formula, and neither is the hidden size or the feed-forward width. The cache is sized by how many key/value heads the model keeps — which is the single lever GQA pulls. ### One spectrum, three names The whole family of attention variants is one dial with two end-stops and a useful middle. Move it by choosing how many query heads share a key/value head. - **Multi-head attention (MHA)** is the original from the 2017 transformer: one key/value head for every query head. Maximum expressiveness, maximum cache. Each head attends with its own view of the past. - **Multi-query attention (MQA)**, proposed by Noam Shazeer in 2019 under the title *"Fast Transformer Decoding: One Write-Head is All You Need"*, is the opposite end: a **single** key/value head shared across *all* query heads. Shazeer's target was exactly the store above — the paper frames slow decoding as "the memory-bandwidth cost of repeatedly loading the large keys and values tensors" and shares them to cut it. The saving is huge; the cost is that every query now attends through the same key/value projection, and quality degrades. - **Grouped-query attention (GQA)**, from Ainslie et al. (2023), is the middle. In the paper's own words, it "divides query heads into G groups, each of which shares a single key head and value head." With G = 1 it *is* MQA; with G equal to the number of query heads it *is* MHA. Anything between is a tunable trade between cache size and quality. The reason the middle exists at all is that MQA turned out to be too aggressive for large models: collapsing to one key/value head loses accuracy and can destabilise training. GQA keeps enough key/value heads to preserve most of the quality while discarding enough to win most of the memory. ### A worked reduction Take Meta's published configuration for **Llama 3.1 70B**: 80 layers, 64 query heads, a head dimension of 128, served in bfloat16 (2 bytes per number). What changes across the three variants is only the key/value head count — 64 for MHA, 8 for the GQA-8 the model actually uses, 1 for MQA. Running the formula for one token, then scaling to a full 131,072-token (128K) session: ``` variant KV heads B/token KiB/token GB @128K vs MHA MHA (64 KV heads) 64 2621440 2560 343.6 1x GQA-8 (8 KV heads) 8 327680 320 42.9 8x MQA (1 KV head) 1 40960 40 5.4 64x ``` Read the last two columns together. Full multi-head attention would demand **344 GB** of KV cache for a single 128K session — more than twice the model's own weights (about 141 GB), so an eight-GPU box with 640 GB of memory could not even hold *one* such session alongside the weights. The GQA-8 the model ships with needs **43 GB**, so the same box serves roughly eleven concurrent sessions. MQA would cut it to 5.4 GB and serve dozens — but at a quality cost the model's authors judged not worth paying. GQA-8 is the point they chose on the dial, and the reduction factor is nothing more mysterious than 64 / 8 = **8x**. The lesson generalises past this one model: the reduction is always `query_heads / kv_heads`, it is free of the parameter count, and it applies per layer and per token, so it compounds with context length. That is why GQA earns its place at the *architecture* level rather than as an inference-time trick. ### You do not have to train it from scratch A practical reason GQA spread so fast is that an existing multi-head model can be *converted* rather than retrained. Ainslie et al. mean-pool the key/value heads within each group to initialise the shared head, then briefly continue training — "uptraining" — to let the model adapt. Their reported cost is small: an uptraining proportion of **α = 0.05**, i.e. about **5% of the original pre-training compute**, recovers most of the quality. The headline finding is that "uptrained GQA achieves quality close to multi-head attention with comparable speed to MQA." A cheap conversion that keeps the quality and buys the memory is close to a free lunch, which is rare enough in this field to explain the near-universal adoption. ## Real-World Applications **The open-weight frontier runs on it.** Grouped-query attention is not a research curiosity; it is the shipping default. Llama 2 70B introduced it to Meta's line, and every Llama 3 and 3.1 size uses it — Llama 3.1 70B's 64-query / 8-key-value layout is the exact configuration worked above. Mistral 7B uses GQA as well, and the ratio it and the Llama models chose (8 key/value heads) has become something close to an industry convention. When a lab publishes a new open model and its config lists fewer `num_key_value_heads` than `num_attention_heads`, that gap *is* GQA, and the difference between the two numbers tells you the memory saving before you read another line. **Sizing an inference deployment.** The most common way a self-hosted model falls over in its first week is that someone provisioned GPUs for the weights and forgot the cache. GQA is what makes the sums survivable: it is the difference between a 70B model needing 344 GB of cache per long-context user (impossible on a single node) and needing 43 GB (eleven users on an 8-GPU box). Any capacity plan — maximum concurrency, maximum context, the long-context surcharge on an API — is computed on the post-GQA cache size, not the multi-head one. **Enabling long context at all.** Vendors advertising 128K, 200K or million-token windows are selling context lengths that would be economically impossible under multi-head attention, because the cache grows linearly with length and GQA divides that whole line by the head ratio. The attention variant and the [context window](https://howaiworks.ai/glossary/context-window) on the spec sheet are the same decision seen from two sides. ## Challenges **The quality cost is real, just usually small and hard to see.** Sharing key/value heads across a group is a genuine loss of representational capacity, and it does not show up on a quick smoke test. Where it surfaces is long-context retrieval — finding a specific fact buried deep in a 100K-token prompt — which is exactly the workload the memory saving was meant to enable. A model can look identical to its multi-head twin on short prompts and quietly do worse on the task you bought the long context for. **Choosing G is a tuning problem with no closed-form answer.** More groups means better quality and a bigger cache; fewer means the reverse. Eight key/value heads is a widely copied convention, not a derived optimum, and the right number depends on model size, target context length and the hardware it will serve on. Picking it well means measuring the quality/memory curve for your own case, not inheriting someone else's ratio. **It shrinks the cache but not the compute.** GQA attacks the memory-bandwidth bottleneck of the decode phase, where a GPU sits idle waiting on bytes. It does nothing for the quadratic arithmetic of attention over a long sequence, which is a separate problem addressed by kernels such as FlashAttention and by sparse or linear-attention schemes. Confusing the two leads teams to reach for GQA when their bottleneck is compute, or for a faster kernel when their bottleneck is cache, and get no improvement either way. **Uptraining is cheap but not nothing.** The 5%-of-pre-training figure is small relative to a full run, but for a frontier model 5% of pre-training compute is still a large, non-trivial job, and the mean-pooling initialisation does not always recover quality cleanly for every architecture. It is a conversion, not a config flag. ## Future Trends The clearest successor is **Multi-head Latent Attention (MLA)**, introduced with DeepSeek-V2. Rather than *dropping* key/value heads the way GQA does, MLA *compresses* them: it projects the keys and values into a shared low-rank latent vector and caches that, reconstructing the per-head keys and values on the fly. DeepSeek reports the compression cuts the KV cache by **93.3%** and lifts maximum generation throughput to **5.76x** relative to their previous 67B model — a deeper reduction than GQA achieves at a comparable head count, which is why later DeepSeek models and several others have adopted it. It is best understood as the next step along the same axis GQA opened: GQA reduces *how many* key/value heads you store, MLA reduces *how large* each stored representation is. The [model-compression](https://howaiworks.ai/glossary/model-compression) view frames it as low-rank factorisation designed into the attention rather than applied to the weights afterwards. Two other directions push on the same cache. **Cross-layer key/value sharing** extends GQA's logic across depth — several layers reusing one set of key/value heads instead of just several query heads within a layer. And **cache quantization** is the orthogonal lever: GQA cuts the head count, storing each value in fewer bits cuts the per-number cost, and the two multiply — a subject that belongs to [quantization](https://howaiworks.ai/glossary/quantization). The through-line for all of them is that the KV cache, not the parameter count, is now the design constraint that shapes attention, and the family of variants around GQA is the industry's response to it. ## Code Example The reduction factor is worth computing directly, because it is the whole argument for GQA in three lines of arithmetic. This script sizes the KV cache per token and per 128K session for the three variants of one 70B-class model, changing nothing but the key/value head count. ```python # KV-cache size per token, and per 128K session, for one 70B-class model # under multi-head, grouped-query and multi-query attention. layers = 80 # Llama 3.1 70B query_heads = 64 head_dim = 128 bytes_per = 2 # bfloat16 context = 131072 # 128K tokens GB = 1e9 def per_token(kv_heads): return 2 * layers * kv_heads * head_dim * bytes_per # 2 = keys + values variants = [ ("MHA (64 KV heads)", 64), ("GQA-8 (8 KV heads)", 8), ("MQA (1 KV head)", 1), ] print(f"{'variant':<20}{'KV heads':>9}{'B/token':>10}{'KiB/token':>11}{'GB @128K':>10}{'vs MHA':>8}") mha = per_token(64) for name, kv in variants: b = per_token(kv) print(f"{name:<20}{kv:>9}{b:>10}{b/1024:>11.0f}{b*context/GB:>10.1f}{mha/b:>7.0f}x") ``` Output: ``` variant KV heads B/token KiB/token GB @128K vs MHA MHA (64 KV heads) 64 2621440 2560 343.6 1x GQA-8 (8 KV heads) 8 327680 320 42.9 8x MQA (1 KV head) 1 40960 40 5.4 64x ``` The `vs MHA` column is just `query_heads / kv_heads`, and every other number scales with it. That single ratio — chosen once when the model is designed — is what separates a model you can serve to a crowd from one you can barely serve to a single user at long context. ## Frequently Asked Questions ### What is grouped-query attention in simple terms? It is a way to shrink the memory a language model carries during generation. Instead of every query head keeping its own key and value vectors, GQA puts query heads into groups and gives each group a single shared key/value head. Fewer stored key/value heads means a smaller KV cache, which is the resource that decides how many users a GPU can serve at long context. ### How is GQA different from multi-query attention? They sit on the same spectrum. Multi-query attention (MQA) shares one key/value head across all query heads — the most aggressive saving, but it costs the most quality. GQA keeps several key/value heads, one per group, landing between MQA and full multi-head attention. The original paper reports that GQA reaches quality close to multi-head attention while being almost as fast as MQA. ### How much does GQA shrink the KV cache? By exactly the ratio of query heads to key/value heads. Llama 3.1 70B has 64 query heads and 8 key/value heads, so its cache is 64/8 = 8x smaller than full multi-head attention would be — 320 KiB per token instead of 2,560 KiB. At 128K context that is 43 GB per session instead of 344 GB. ### Which models use grouped-query attention? Nearly every open-weight frontier model since 2023: Llama 2 70B, all Llama 3 and 3.1 sizes, and Mistral 7B among many others. It has become the default attention variant precisely because it cuts serving cost with little measurable quality loss. ### What is Multi-head Latent Attention (MLA)? MLA, introduced with DeepSeek-V2, is a newer alternative that compresses keys and values into a shared low-rank latent vector rather than dropping heads. DeepSeek reports it cuts the KV cache by 93.3% and lifts maximum generation throughput to 5.76x versus their previous 67B model — a larger reduction than GQA at a given head count. ## Related ### Related terms - [KV Cache](https://howaiworks.ai/glossary/kv-cache) - [Attention Mechanism](https://howaiworks.ai/glossary/attention-mechanism) - [Self-Attention](https://howaiworks.ai/glossary/self-attention) - [Transformer](https://howaiworks.ai/glossary/transformer) - [Inference Optimization](https://howaiworks.ai/glossary/inference-optimization) - [Model Compression](https://howaiworks.ai/glossary/model-compression) --- Source: https://howaiworks.ai/glossary/grouped-query-attention — HowAIWorks.ai