Picture a chatbot mid-conversation, holding a growing stack of sticky notes — one for every word anyone has typed so far — so it never has to re-read the whole chat from scratch to write its next word. That stack is real, it lives in the AI’s memory chip, and it is the reason long conversations get slow, expensive, or refuse to load another user. MQA vs. GQA vs. MLA names the three main engineering answers to that one problem: how do you keep enough of the stack to stay accurate, while throwing away enough of it to stay fast?
This article walks through all three techniques in order — Multi-Query Attention, Grouped-Query Attention, and Multi-Head Latent Attention — with a plain-English explanation of each, honest credit for who built what and when, and a worked cache-size comparison so the trade-offs are numbers, not vibes. The de-hype note up front: these three techniques all attack the same decode-time memory bottleneck. They do nothing about the separate prefill compute cost of reading a long prompt in the first place — that’s a different wall, tackled by different tools.
Two Different “Slow”: Why This Is a Decode-Time Problem
Large language models get expensive in two distinct moments, and mixing them up is the single most common mistake in explaining this space. The first moment is prefill — the model reads your entire prompt at once, comparing every word to every other word, which costs compute that grows with the square of the prompt length. The second moment is decode — the model generates its reply one word at a time, and for every new word it has to reference the Key and Value vectors it stored for every word that came before. That storage is the KV cache, and it is a memory-bandwidth problem, not a compute problem: the GPU has plenty of raw arithmetic power to spare, but shuttling a huge cache in and out of fast memory for every single generated word becomes the bottleneck. (LIG’s KV-cache explainer covers that mechanism in more depth, and why long context is expensive covers the prefill half of the story.)
MQA, GQA, and MLA all live entirely on the decode side of that split. None of them speed up reading your prompt. What they change is how much has to be remembered — and re-fetched — for every word the model writes afterward.
The Baseline: Multi-Head Attention Remembers Everything
Standard Multi-Head Attention (MHA) — the design the original Transformer shipped with — gives every attention head its own private set of Key and Value vectors. If a model has 32 heads, it keeps 32 separate copies of “what happened so far,” one per head, for every layer, for every token. That’s excellent for quality: each head gets its own undiluted view of the conversation. It’s expensive for memory: the cache size scales directly with the number of heads, and doubling the heads roughly doubles the bill. MHA is the “no compromises” starting point every other technique below is measured against.
MQA: One Shared Memory for Every Head (Shazeer, 2019)
Multi-Query Attention (MQA) was introduced by Noam Shazeer at Google in 2019, in a paper bluntly titled “Fast Transformer Decoding: One Write-Head is All You Need” (arXiv:1911.02150). The idea is the most aggressive cut available: instead of 32 separate Key-Value sets, all 32 heads share exactly one. The cache shrinks by roughly the number of heads you’d otherwise be storing — a 32-head model’s cache drops to about 1/32nd of MHA’s size, independent of how many heads it has.
The cost is quality. When every head is forced to read from the same shared memory, each one loses its individual “point of view” on the conversation, and benchmarks show a real, measurable accuracy hit compared to full attention. MQA proved the concept — that heads don’t strictly need private caches — but few production models shipped it in its purest form, because the quality trade was steeper than most teams wanted to accept.
GQA: The Grouped Compromise That Became the Standard (Ainslie et al., 2023)
Grouped-Query Attention (GQA) came from Ainslie et al. at Google Research in 2023 (arXiv:2305.13245, published at EMNLP). It sits deliberately between the two extremes: instead of every head sharing one cache (MQA) or every head keeping its own (MHA), heads are split into a small number of groups, and every head within a group shares one Key-Value set.
This turned out to be the sweet spot, and it’s not theoretical — it’s what shipped. Checking real, published model configurations: Mistral 7B uses 32 query heads split into 8 groups (arXiv:2310.06825). Llama 3 8B uses the identical shape — 32 query heads, 8 KV groups. Llama 2 70B uses 64 query heads grouped down to 8 KV heads (arXiv:2307.09288) — worth noting that Llama 2’s smaller 7B and 13B variants stuck with plain MHA; GQA was reserved for the largest model, where the memory bill hurts most. That 8-group pattern recurring across three different labs’ model families is a good signal GQA isn’t a fad — enough teams converged on it independently that it’s fair to call it the current industry default for open-weight models.
MLA: Compressing Instead of Sharing (DeepSeek, 2024)
Multi-Head Latent Attention (MLA), introduced by DeepSeek in the DeepSeek-V2 technical report in 2024 (arXiv:2405.04434), takes a genuinely different approach rather than a further point along the MQA-to-GQA sharing spectrum. Instead of having heads share raw Keys and Values, MLA compresses each token’s Key and Value information down into one small “latent” vector, caches only that, and reconstructs the full per-head detail on demand when it’s needed — the zip-and-unzip analogy is close to literal here.
Why bother with the extra reconstruction step? Because MQA and GQA both sit on the same line: the more you compress, the more quality you lose. DeepSeek’s reported results claim MLA doesn’t just pick a different point on that line, it bends the curve — a cache footprint in the same range as GQA under heavy compression, but with output quality DeepSeek reports as matching or exceeding full, uncompressed MHA. That’s the whole reason MLA earned its own name rather than being filed as “DeepSeek’s version of GQA.” One number attached to it needs a careful caveat: DeepSeek reports MLA’s cache as 93.3% smaller than the KV cache of DeepSeek’s own previous 67B dense model — a real, verifiable result from their own paper, but a comparison against their own prior architecture, not a universal “93% smaller than any Transformer” claim. It’s also worth noting MLA’s reconstruction math needs more custom engineering to serve efficiently than GQA’s simple head-grouping — inference engines like vLLM and SGLang added MLA support only after DeepSeek’s models made it worth building, so its footprint outside DeepSeek’s own stack is narrower than GQA’s, as of this writing.
The Cache-Size Math, Worked Through
The formula behind all of this is simpler than it looks: a KV cache’s size is proportional to 2 (Key + Value) × layers × KV-heads × head-dimension × sequence-length × bytes-per-number. Only the KV-heads term changes across MHA, MQA, and GQA — everything else about the model stays fixed. That means the savings scale is entirely about how many independent Key-Value sets you’re storing.
Take one consistent hypothetical model to see it in bytes: 32 layers, 32 attention heads, a head dimension of 128, a 4,096-token conversation, and 16-bit (2-byte) storage per number — architecture numbers deliberately close to real 7-8B-class open models.
| Technique | KV-heads stored | Cache size (this model, 4K tokens) | Relative to MHA |
|---|---|---|---|
| MHA (baseline) | 32 (all of them) | ~2.1 GB | 1× (100%) |
| GQA, 8 groups | 8 | ~537 MB | 4× smaller (25%) |
| MQA | 1 | ~67 MB | 32× smaller (~3%) |
| MLA (DeepSeek’s own reported figure) | compressed latent, not a head count | not directly comparable — DeepSeek reports ~93.3% smaller than their own 67B MHA baseline, a different model shape entirely | see caveat above |
That last row is why the two families don’t belong in a single fair table cell: MQA and GQA save memory by storing fewer full-size Key-Value sets, so their savings are a clean ratio you can compute from head counts alone. MLA saves memory by storing a compressed representation of every token, so its savings depend on the latent dimension DeepSeek chose for its own architecture — comparable in spirit, not in formula. Sharing and compression are different tricks aimed at the same wall, and it’s worth resisting the urge to rank them on one number.
What “Best” Actually Depends On
There’s no universal winner — only the right tool for a given constraint. A memory-starved team might accept MQA’s steepest quality cost for its biggest cut. A team that wants the safe, well-supported default gets GQA, which is why Llama and Mistral both ship with it — the serving frameworks, quantization tools, and fine-tuning recipes are built around it. A team operating at DeepSeek’s scale, with the engineering to support MLA’s reconstruction math, gets DeepSeek’s claimed edge on quality at a comparable cache size.
The broader habit worth keeping from all three: whenever an efficiency claim shows up as a bare percentage — “93% smaller,” “32× less memory” — ask compared to what baseline, then ask what did it cost in quality. A savings number with no baseline is marketing. One with no quality figure is telling half the story.
❓ Frequently Asked Questions
What is the difference between MQA, GQA, and MLA? All three shrink the KV cache an AI model relies on while generating text. MQA (Shazeer, 2019) makes every attention head share one Key-Value set — the biggest memory savings, the biggest quality trade-off. GQA (Ainslie et al., 2023) groups heads so each group shares a set — the balanced middle ground that became the industry standard. MLA (DeepSeek, 2024) compresses Keys and Values into a small latent vector and reconstructs them on demand — a different mechanism aiming to beat the memory-versus-quality trade-off, not just pick a point on it.
Do MQA, GQA, and MLA speed up reading a long prompt? No. That’s a separate cost — the O(n²) prefill compute wall, tackled by different techniques like sparse attention and FlashAttention. MQA, GQA, and MLA only affect the decode-time KV cache, the memory used while generating a reply one word at a time.
Did DeepSeek invent shrinking the KV cache? No. Google introduced MQA in 2019 and GQA in 2023 — years before DeepSeek’s MLA in 2024. DeepSeek’s real contribution is MLA specifically: a compression-based approach, not the broader idea of reducing KV-cache memory, which predates it.
Is GQA better than MLA? Neither is universally “better” — they’re different trade-offs. GQA is simpler to implement and has wider support across serving software like vLLM and llama.cpp, which is why it’s the default in Llama and Mistral. MLA claims a better memory-versus-quality curve in DeepSeek’s own benchmarks, but needs more specialized engineering to serve efficiently and has narrower deployment outside DeepSeek’s own models.
Why does a bigger KV cache matter for regular users? A larger cache means a GPU can serve fewer simultaneous conversations, or a single long conversation runs slower and costs more to host. Shrinking the cache is one of the main reasons some AI chatbots stay responsive and affordable even as context windows and user counts grow.
📚 Further Reading
- How AI Beats the “n² Wall”: The Attention-Efficiency Playbook — the cornerstone guide to every technique fighting attention’s cost, including where MQA/GQA/MLA fit in the bigger picture.
- What Is a KV Cache, and Why Does Long Context Cost So Much Memory? — a deeper look at the exact mechanism these three techniques are all shrinking.
- Why Long Context Is Expensive — the prefill-side compute cost, the other half of the “n² wall.”
- Multi-Head Latent Attention Explained — a dedicated deep dive on DeepSeek’s compression mechanism.
📚 Recommended Resources
Disclosure: some links below are Amazon affiliate links. If you buy through them, I earn a small commission — at no extra cost to you — which helps keep this site running.
Build a Large Language Model (From Scratch)
Sebastian Raschka · Book · ~$50 on Amazon
If the cache-size math in this article made you want to see it running in real code, this is the book — it walks through implementing attention, the KV cache, and a working Transformer from first principles, so the concepts stop being diagrams and start being lines you can run yourself.
Natural Language Processing with Transformers
Lewis Tunstall, Leandro von Werra, Thomas Wolf · Book · ~$60 on Amazon
Written by the Hugging Face team, this is the practical follow-up once you understand why MQA, GQA, and MLA exist — how to actually load, fine-tune, and deploy the Transformer models that use them.
Hands-On Machine Learning with Scikit-Learn, Keras & TensorFlow
Aurélien Géron · Book · ~$70 on Amazon
The best all-around reference for going from machine-learning fundamentals to modern deep learning. A solid foundation if you want to understand the layers below attention before tackling cache-optimization techniques directly.