GQA vs MHA gets treated as a single yes-or-no toggle: does a model use grouped-query attention or not. That framing hides the number that actually decides how much VRAM a model needs for its KV cache. Llama 3.1 8B and Gemma 2 9B are both correctly described as "GQA" models, but Llama 3.1 cuts its KV cache 4x versus an equivalent MHA model while Gemma 2 cuts it only 2x. We pulled the config.json for seven released models to show the gap and where each one actually falls. The entire difference comes down to one field: num_key_value_heads.
TL;DR: GQA vs MHA KV Cache Memory Savings
- Not a fixed ratio: GQA cuts KV cache 4x on Llama 3.1 8B and Mistral 7B v0.3, 7x on Qwen2.5 7B, and 8x on Llama 3.1 70B, but only 2x on Gemma 2 9B and 27B, per each model's `config.json`.
- The formula: KV cache bytes/token = 2 x num_key_value_heads x head_dim x layers x bytes-per-value. Only
num_key_value_headsdiffers between a model's MHA and GQA versions. - Worked example: an 8B Llama-shaped model at 32K context needs about 16GB of KV cache per user under MHA versus about 4GB under GQA, roughly 7 vs 30 concurrent users on one 141GB H200.
- Spheron: H200 SXM5 on-demand runs $5.92/hr as of 10 Sep 2026. Rent an H200 to test it →
MHA, MQA, and GQA: One Diagram, Three Head-Sharing Patterns
All three mechanisms compute the same number of query heads. What differs is how many distinct key/value heads those queries read from.
MHA, every query head gets its own key/value head:
Q1 Q2 Q3 Q4 ... Q32
| | | | |
K1 K2 K3 K4 ... K32GQA, queries are split into groups that share one key/value head (Llama 3.1 8B: 32 queries, 8 groups of 4):
Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8 ... Q29 Q30 Q31 Q32
\ | / \ | / \ | /
K1 K2 ... K8MQA, every query head reads from a single shared key/value head:
Q1 Q2 Q3 ... Q32
\ | / /
K1Multi-query attention came first. Noam Shazeer introduced it in 2019 as a way to speed up decoder inference by writing one key/value head instead of one per query head, and it was later adopted by models including PaLM and Falcon.
The GQA paper reports that GQA lands close to MQA's speed while achieving quality closer to full MHA than MQA achieves.
The KV Cache Math: Heads x Head Dim x Layers x Tokens
The bytes a model has to store in its KV cache for one token, at one layer, come down to one formula:
KV bytes per token per layer = 2 x num_key_value_heads x head_dim x bytes_per_valueThe 2 covers the K tensor and the V tensor. bytes_per_value is 2 for BF16/FP16, 1 for FP8. Multiply by the number of transformer layers to get bytes per token for the whole model, then by context length to get bytes per user, then by concurrent users to get total KV cache demand.
num_attention_heads (query heads) never appears in this formula. That's the entire point of GQA: query-side compute is untouched, only the stored K/V footprint shrinks. For Llama 3.1 8B, with num_key_value_heads = 8 and head_dim = 128: 2 x 8 x 128 x 2 = 4,096 bytes per token per layer. Across its 32 layers, that's 131,072 bytes, 128 KiB, per token for one user's entire KV cache.
This formula only governs what gets stored, not how fast it gets read back. Speeding up the read side is a separate lever: our FlashAttention breakdown covers how that works and why it doesn't shrink the cache itself.
GQA vs MHA KV Cache Savings Across Llama, Mistral, Qwen, and Gemma
Pulling num_attention_heads, num_key_value_heads, and head_dim straight from each model's published config.json shows the reduction ratio is not a property of "using GQA." It is a property of the specific numbers a lab chose:
| Model | Query Heads | KV Heads | Reduction vs MHA | head_dim | KV Bytes/Token/Layer (BF16) |
|---|---|---|---|---|---|
| Llama 2 7B (MHA) | 32 | 32 | 1x (no GQA) | 128 | 16,384 |
| Llama 3.1 8B | 32 | 8 | 4x | 128 | 4,096 |
| Mistral 7B v0.3 | 32 | 8 | 4x | 128 | 4,096 |
| Qwen2.5 7B | 28 | 4 | 7x | 128 | 2,048 |
| Llama 3.1 70B | 64 | 8 | 8x | 128 | 4,096 |
| Gemma 2 9B | 16 | 8 | 2x | 256 | 8,192 |
| Gemma 2 27B | 32 | 16 | 2x | 128 | 8,192 |
Llama 2 7B is the baseline worth noticing first: its `config.json` lists num_attention_heads: 32 and num_key_value_heads: 32, the same value twice. That's plain MHA, not GQA at all, despite Llama 2 shipping in the same era GQA was becoming standard. The GQA-vs-MHA question genuinely depends on checking the specific checkpoint, not assuming a model family is uniform.
The comparison we find more useful is Qwen2.5 7B against Llama 3.1 8B. Both are roughly 7-8B parameter models with head_dim = 128, and both are correctly labeled GQA. But Qwen2.5 7B caches 2,048 bytes per token per layer against Llama 3.1 8B's 4,096, half as much, because Qwen went with 4 KV heads instead of 8. "This model uses GQA" tells you nothing about which of these two it is; you have to read the field yourself.
Why Gemma's GQA Saves Less Than Llama's (2x vs 8x)
Gemma 2's smaller reduction ratio isn't a technical ceiling, it's a deliberate choice the Gemma 2 team made and published their reasoning for. Both Gemma 2 9B and 27B use num_groups = 2, meaning every KV head serves only 2 query heads, versus 4 query heads per KV head on Llama 3.1 8B or 7 on Qwen2.5 7B.
The Gemma 2 technical report states the tradeoff directly: "We choose GQA since it requires fewer parameters and is faster at inference time."
The practical read: a 2x-reduction "GQA" model is not under-optimized relative to a 4x or 8x one. It's a different point on the same quality-versus-memory curve, and the only way to know where a specific checkpoint sits is to read num_key_value_heads and head_dim out of its config.json, exactly as the table above did.
Worked Number: How Many More Users Fit on One GPU
Here's the sanity check we run whenever we size a GPU for a new model. Take Llama 3.1 8B's GQA config (32 query heads, 8 KV heads, head_dim 128, 32 layers) against a same-shaped MHA model (Llama 2 7B's 32/32/128/32 config, which is architecturally almost identical apart from the KV head count). At 32K context, a realistic length for RAG or multi-turn agent workloads:
| Attention type | KV heads | KV cache per user @ 32K context | Concurrent users on 141GB H200 |
|---|---|---|---|
| MHA (Llama-2-7B-shaped) | 32 | 16 GiB | ~7 |
| GQA (Llama 3.1 8B) | 8 | 4 GiB | ~30 |
Those 16 GiB and 4 GiB figures come straight from the formula above: 4,096 bytes x 32 layers x 32,768 tokens for GQA, and 16,384 bytes x 32 layers x 32,768 tokens for MHA. An 8B model's weights take roughly 15GB of VRAM in BF16, leaving roughly 120GB of a 141GB H200 SXM5 for KV cache and runtime overhead after that. Divide that budget by the per-user cache size and MHA fits about 7 concurrent users at this context length; GQA fits about 30, a roughly 4x jump that tracks the 4x KV head reduction almost exactly.
Spheron aggregates on-demand pricing across 5+ providers with per-minute billing and no minimum rental period, and an on-demand H200 SXM5 runs $5.92/hr as of 10 Sep 2026. If you want to check this VRAM budget yourself, provisioning a GPU and running vllm serve with --max-model-len 32768 against both a GQA and an MHA checkpoint costs a few dollars of GPU time, not a capital purchase; Spheron's deployment docs cover the provisioning steps. GPU pricing moves with availability, so check current rates before locking in a capacity plan.
Pricing fluctuates based on GPU availability. Spheron rates above are live as of 10 Sep 2026; other providers reflect their most recent published rates and may have changed. Check current GPU pricing → for live rates.
For deployment specifics on long-context H200 setups, including multi-model and NVLink cluster configuration, see the H200 deployment guide.
Where GQA Runs Out: MLA and Other Next Steps
GQA is a fixed multiplier: pick 4 KV heads instead of 32 and the cache shrinks 8x, permanently, for every request. At short context that 8x is plenty. At 128K+ context, even an 8x-reduced cache can still be tens of gigabytes per user, which is why newer architectures push further than head-sharing alone.
Multi-Head Latent Attention (MLA), used by DeepSeek V3 and the Kimi K2 series, replaces per-head K/V storage entirely with a single compressed latent vector, typically cutting KV cache by another order of magnitude beyond what GQA reaches on a comparably sized model. It's worth reading if your target model's context length makes even a 4x-8x GQA saving insufficient.
KV cache compression and attention architecture are separate, stackable levers, not competitors. Whatever GQA leaves in the cache can be quantized further; Google's TurboQuant compresses an existing KV cache after the fact rather than changing how many heads get stored in the first place. GQA determines the size of the memory problem; the read-side optimizations covered above determine how fast you can compute over it.
Once a KV cache is as small as GQA (or MLA) can make it, the remaining bottleneck for very long contexts becomes fitting a single sequence across multiple GPUs at all. That's the problem ring attention and tree attention address, splitting one sequence's attention computation across a GPU cluster rather than trying to fit it on one card.
None of this is something a GPU host can change. Whether a checkpoint uses MHA, GQA, or MLA is fixed at pretraining time by the model's authors; no amount of extra VRAM retrofits GQA onto a model trained with full MHA. The only lever a host controls is how much memory you can rent to serve whatever architecture you're stuck with, and how cheaply you can test the real numbers before committing to a GPU size.
Whether a model ships with MHA, GQA, or MLA is fixed long before it reaches a GPU. Renting the right amount of VRAM for the KV cache it actually needs is the part still in your control.
Frequently Asked Questions
Multi-head attention (MHA) gives every query head its own key and value head, so a 32-head model caches 32 separate K/V pairs per token per layer. Grouped-query attention (GQA) splits query heads into groups that share one key/value head each, so the same 32-head model might cache only 8 K/V pairs, a 4x cut. The query-side compute is unchanged; only the number of stored, and later reloaded, key/value heads changes.
It depends entirely on num_key_value_heads, not on the label 'GQA' by itself. Llama 3.1 8B and Mistral 7B v0.3 both go from 32 query heads to 8 KV heads, a 4x reduction. Qwen2.5 7B goes from 28 to 4, a 7x reduction. Llama 3.1 70B goes from 64 to 8, an 8x reduction. Gemma 2 9B and 27B both only go 2x (16 to 8, and 32 to 16). Two models can both say 'GQA' in their architecture description and differ by 4x in actual KV cache savings.
num_key_value_heads sets how many distinct key/value head groups the model stores and computes, separate from num_attention_heads, which sets the number of query heads. When the two fields are equal, the model is plain multi-head attention. When num_key_value_heads is 1, it's multi-query attention. Any value in between is grouped-query attention, and the ratio between the two fields is exactly the KV cache reduction factor versus an equivalent MHA model.
The GQA paper's uptraining experiments report perplexity close to full MHA while cutting inference latency toward multi-query attention speeds, which is the standard justification labs give for shipping GQA instead of MHA.
Yes, this is called uptraining. The original GQA paper takes existing multi-head checkpoints and continues training them with mean-pooled key/value heads for a small additional compute budget, about 5% of the model's original pretraining compute, rather than requiring a full retrain from scratch. This is how several GQA-family models were derived from earlier MHA checkpoints instead of being pretrained as GQA from the first step.
