Definition
Grouped-query attention (GQA) is the reason a 70B model can serve roughly eight times as many users on the same GPU as it could with textbook attention, while answering almost as well. In a standard transformer, every one of the model's attention heads keeps its own running notes on the conversation — a key and a value vector per token, per layer, held in memory for as long as the session lives. That pile of notes is the KV cache, and at long context it, not the model's weights, is what fills up the GPU. GQA shrinks it by a simple structural change: instead of every query head owning a private key/value head, query heads are sorted into groups, and each group shares one key/value head.
The number that follows from that change is exact and durable: the cache shrinks by the ratio of query heads to key/value heads. Cut 64 key/value heads down to 8 and the cache is eight times smaller — for one architectural decision that leaves the parameter count essentially unchanged and, in careful measurements, barely moves output quality. That trade is why GQA is not an exotic optimisation but the default in nearly every open model shipped since 2023.
How It Works
The thing GQA is attacking
To generate each new word, a transformer's self-attention lets the current position look back at every earlier one. To avoid re-reading the whole history at every step, the model stores the key and value vectors it already computed. The size of that store, for a standard model, is:
bytes per token = 2 (keys and values) x layers x KV heads x head dimension x bytes per number
The quantity to stare at is KV heads. The number of query heads is not in the formula, and neither is the hidden size or the feed-forward width. The cache is sized by how many key/value heads the model keeps — which is the single lever GQA pulls.
One spectrum, three names
The whole family of attention variants is one dial with two end-stops and a useful middle. Move it by choosing how many query heads share a key/value head.
- Multi-head attention (MHA) is the original from the 2017 transformer: one key/value head for every query head. Maximum expressiveness, maximum cache. Each head attends with its own view of the past.
- Multi-query attention (MQA), proposed by Noam Shazeer in 2019 under the title "Fast Transformer Decoding: One Write-Head is All You Need", is the opposite end: a single key/value head shared across all query heads. Shazeer's target was exactly the store above — the paper frames slow decoding as "the memory-bandwidth cost of repeatedly loading the large keys and values tensors" and shares them to cut it. The saving is huge; the cost is that every query now attends through the same key/value projection, and quality degrades.
- Grouped-query attention (GQA), from Ainslie et al. (2023), is the middle. In the paper's own words, it "divides query heads into G groups, each of which shares a single key head and value head." With G = 1 it is MQA; with G equal to the number of query heads it is MHA. Anything between is a tunable trade between cache size and quality.
The reason the middle exists at all is that MQA turned out to be too aggressive for large models: collapsing to one key/value head loses accuracy and can destabilise training. GQA keeps enough key/value heads to preserve most of the quality while discarding enough to win most of the memory.
A worked reduction
Take Meta's published configuration for Llama 3.1 70B: 80 layers, 64 query heads, a head dimension of 128, served in bfloat16 (2 bytes per number). What changes across the three variants is only the key/value head count — 64 for MHA, 8 for the GQA-8 the model actually uses, 1 for MQA. Running the formula for one token, then scaling to a full 131,072-token (128K) session:
variant KV heads B/token KiB/token GB @128K vs MHA
MHA (64 KV heads) 64 2621440 2560 343.6 1x
GQA-8 (8 KV heads) 8 327680 320 42.9 8x
MQA (1 KV head) 1 40960 40 5.4 64x
Read the last two columns together. Full multi-head attention would demand 344 GB of KV cache for a single 128K session — more than twice the model's own weights (about 141 GB), so an eight-GPU box with 640 GB of memory could not even hold one such session alongside the weights. The GQA-8 the model ships with needs 43 GB, so the same box serves roughly eleven concurrent sessions. MQA would cut it to 5.4 GB and serve dozens — but at a quality cost the model's authors judged not worth paying. GQA-8 is the point they chose on the dial, and the reduction factor is nothing more mysterious than 64 / 8 = 8x.
The lesson generalises past this one model: the reduction is always query_heads / kv_heads, it is
free of the parameter count, and it applies per layer and per token, so it compounds with context
length. That is why GQA earns its place at the architecture level rather than as an
inference-time trick.
You do not have to train it from scratch
A practical reason GQA spread so fast is that an existing multi-head model can be converted rather than retrained. Ainslie et al. mean-pool the key/value heads within each group to initialise the shared head, then briefly continue training — "uptraining" — to let the model adapt. Their reported cost is small: an uptraining proportion of α = 0.05, i.e. about 5% of the original pre-training compute, recovers most of the quality. The headline finding is that "uptrained GQA achieves quality close to multi-head attention with comparable speed to MQA." A cheap conversion that keeps the quality and buys the memory is close to a free lunch, which is rare enough in this field to explain the near-universal adoption.
Real-World Applications
The open-weight frontier runs on it. Grouped-query attention is not a research curiosity; it is
the shipping default. Llama 2 70B introduced it to Meta's line, and every Llama 3 and 3.1 size uses
it — Llama 3.1 70B's 64-query / 8-key-value layout is the exact configuration worked above. Mistral
7B uses GQA as well, and the ratio it and the Llama models chose (8 key/value heads) has become
something close to an industry convention. When a lab publishes a new open model and its config
lists fewer num_key_value_heads than num_attention_heads, that gap is GQA, and the difference
between the two numbers tells you the memory saving before you read another line.
Sizing an inference deployment. The most common way a self-hosted model falls over in its first week is that someone provisioned GPUs for the weights and forgot the cache. GQA is what makes the sums survivable: it is the difference between a 70B model needing 344 GB of cache per long-context user (impossible on a single node) and needing 43 GB (eleven users on an 8-GPU box). Any capacity plan — maximum concurrency, maximum context, the long-context surcharge on an API — is computed on the post-GQA cache size, not the multi-head one.
Enabling long context at all. Vendors advertising 128K, 200K or million-token windows are selling context lengths that would be economically impossible under multi-head attention, because the cache grows linearly with length and GQA divides that whole line by the head ratio. The attention variant and the context window on the spec sheet are the same decision seen from two sides.
Challenges
The quality cost is real, just usually small and hard to see. Sharing key/value heads across a group is a genuine loss of representational capacity, and it does not show up on a quick smoke test. Where it surfaces is long-context retrieval — finding a specific fact buried deep in a 100K-token prompt — which is exactly the workload the memory saving was meant to enable. A model can look identical to its multi-head twin on short prompts and quietly do worse on the task you bought the long context for.
Choosing G is a tuning problem with no closed-form answer. More groups means better quality and a bigger cache; fewer means the reverse. Eight key/value heads is a widely copied convention, not a derived optimum, and the right number depends on model size, target context length and the hardware it will serve on. Picking it well means measuring the quality/memory curve for your own case, not inheriting someone else's ratio.
It shrinks the cache but not the compute. GQA attacks the memory-bandwidth bottleneck of the decode phase, where a GPU sits idle waiting on bytes. It does nothing for the quadratic arithmetic of attention over a long sequence, which is a separate problem addressed by kernels such as FlashAttention and by sparse or linear-attention schemes. Confusing the two leads teams to reach for GQA when their bottleneck is compute, or for a faster kernel when their bottleneck is cache, and get no improvement either way.
Uptraining is cheap but not nothing. The 5%-of-pre-training figure is small relative to a full run, but for a frontier model 5% of pre-training compute is still a large, non-trivial job, and the mean-pooling initialisation does not always recover quality cleanly for every architecture. It is a conversion, not a config flag.
Future Trends
The clearest successor is Multi-head Latent Attention (MLA), introduced with DeepSeek-V2. Rather than dropping key/value heads the way GQA does, MLA compresses them: it projects the keys and values into a shared low-rank latent vector and caches that, reconstructing the per-head keys and values on the fly. DeepSeek reports the compression cuts the KV cache by 93.3% and lifts maximum generation throughput to 5.76x relative to their previous 67B model — a deeper reduction than GQA achieves at a comparable head count, which is why later DeepSeek models and several others have adopted it. It is best understood as the next step along the same axis GQA opened: GQA reduces how many key/value heads you store, MLA reduces how large each stored representation is. The model-compression view frames it as low-rank factorisation designed into the attention rather than applied to the weights afterwards.
Two other directions push on the same cache. Cross-layer key/value sharing extends GQA's logic across depth — several layers reusing one set of key/value heads instead of just several query heads within a layer. And cache quantization is the orthogonal lever: GQA cuts the head count, storing each value in fewer bits cuts the per-number cost, and the two multiply — a subject that belongs to quantization. The through-line for all of them is that the KV cache, not the parameter count, is now the design constraint that shapes attention, and the family of variants around GQA is the industry's response to it.
Code Example
The reduction factor is worth computing directly, because it is the whole argument for GQA in three lines of arithmetic. This script sizes the KV cache per token and per 128K session for the three variants of one 70B-class model, changing nothing but the key/value head count.
# KV-cache size per token, and per 128K session, for one 70B-class model
# under multi-head, grouped-query and multi-query attention.
layers = 80 # Llama 3.1 70B
query_heads = 64
head_dim = 128
bytes_per = 2 # bfloat16
context = 131072 # 128K tokens
GB = 1e9
def per_token(kv_heads):
return 2 * layers * kv_heads * head_dim * bytes_per # 2 = keys + values
variants = [
("MHA (64 KV heads)", 64),
("GQA-8 (8 KV heads)", 8),
("MQA (1 KV head)", 1),
]
print(f"{'variant':<20}{'KV heads':>9}{'B/token':>10}{'KiB/token':>11}{'GB @128K':>10}{'vs MHA':>8}")
mha = per_token(64)
for name, kv in variants:
b = per_token(kv)
print(f"{name:<20}{kv:>9}{b:>10}{b/1024:>11.0f}{b*context/GB:>10.1f}{mha/b:>7.0f}x")
Output:
variant KV heads B/token KiB/token GB @128K vs MHA
MHA (64 KV heads) 64 2621440 2560 343.6 1x
GQA-8 (8 KV heads) 8 327680 320 42.9 8x
MQA (1 KV head) 1 40960 40 5.4 64x
The vs MHA column is just query_heads / kv_heads, and every other number scales with it. That
single ratio — chosen once when the model is designed — is what separates a model you can serve to a
crowd from one you can barely serve to a single user at long context.