Every time a new AI model brags about reading a million words without breaking a sweat, there’s a decent chance a state-space model is doing some of the work — or, just as often, it’s still a Transformer with a few clever tricks bolted on. The Mamba vs Transformer debate sounds like a rivalry with a winner, but it’s really a trade-off, and reading it honestly means understanding what each architecture is actually built to remember.
This piece scores that trade-off the way a Measure piece should: where Mamba and its state-space relatives genuinely win (throughput, memory, streaming), where the plain old Transformer still wins (exact recall, copying long stretches word-for-word), and why — despite the 2023–24 hype — nobody actually crowned a “Transformer-killer.” The architectures that shipped into real products are hybrids of both.
A Different Way to Remember a Conversation
A Transformer’s attention mechanism works like a meeting where, before saying anything, every participant re-reads the entire transcript so far — comparing what they’re about to say against everything anyone has said. That’s what makes attention so good at using context, but it’s also why the cost explodes: double the transcript, and the comparison work quadruples — the O(n²) wall this whole hub is built around. Worse, as the model generates each new word, it has to keep a running store of every past word’s details on hand — the KV cache — and that store keeps growing for as long as the conversation runs.
State-space models (SSMs) take a genuinely different approach, borrowed from classical control theory and old-school recurrent networks. Instead of re-reading the whole transcript at every step, an SSM carries forward one fixed-size “notebook” — a compressed summary of everything relevant so far — and updates it a token at a time. Nothing about the notebook grows as the conversation gets longer. That single design choice is the source of everything SSMs are good at, and everything they struggle with.
From S4 to Mamba: Making the Compression Smart
The modern SSM line starts with S4 (“Structured State Spaces,” Gu, Goel, and Ré, ICLR 2022). S4’s contribution was making the notebook’s update math (formally, matrices called A, B, and C) structured enough — using something called HiPPO initialization — that it could capture long-range dependencies without the numbers blowing up or fading to nothing, the classic failure mode of older recurrent networks. It did well on long-sequence benchmarks. But S4’s update rule was fixed: it jotted things into the notebook the same way regardless of what was actually being said, with no ability to decide “this part matters, skip the rest.”
Mamba (Gu and Dao, December 2023, arXiv:2312.00752) fixed exactly that. Mamba made the notebook’s update rule selective — a function of the input itself, so the model can dynamically decide, token by token, what’s worth remembering and what to let fade. That’s much closer to what attention does implicitly by weighting some words more than others. The catch is that selectivity breaks the trick that let S4 train in parallel like a convolution, so Gu and Dao designed a hardware-aware parallel scan that exploits GPU memory hierarchy to keep training fast anyway. Mamba drops attention and the usual MLP blocks entirely. In the paper’s own benchmarks, it reports roughly 5× higher inference throughput than same-size Transformers, with a 3-billion-parameter Mamba matching Transformers roughly twice its size on language modeling.
Mamba-2 and a Surprising Piece of Math
Mamba-2 (Dao and Gu, May 2024, arXiv:2405.21060) pushed the architecture further, but its most interesting contribution isn’t a speed number — it’s a proof. Called State Space Duality (SSD), it formally shows, via structured semiseparable matrices, that selective SSMs and a form of linear attention are the same family of computation, just expressed two different ways. That’s a genuinely elegant result: the “reformulate the softmax into a fast approximation” line of research and the “replace attention with a recurrent state” line of research turn out to converge on the same math. Mamba-2’s core layer is also 2–8× faster than Mamba-1’s while staying competitive on quality.
Running in parallel to the Mamba lineage is RWKV (“Receptance Weighted Key Value”), an independently developed, fully attention-free architecture that trains in parallel like a Transformer but runs as a constant-memory recurrent network at inference — no KV cache, unbounded context in principle. As of 2026 it has iterated through RWKV-7 (“Goose,” up to 13 billion parameters), with RWKV-8 reportedly in development — worth checking current status before citing exact numbers, since this corner of the field moves fast.
The Honest Scorecard: Throughput and Memory
Here’s where SSMs unambiguously win, measured the way this hub insists on measuring things — against a stated baseline, not a headline.
| Transformer (full attention) | Mamba / SSM | |
|---|---|---|
| Training cost vs. sequence length | Quadratic — O(n²) | Linear — O(n) |
| Inference memory | KV cache grows with every generated token | Fixed-size hidden state — flat, constant |
| Very long streaming input | Cache balloons, gets expensive fast | Cheap in principle, near-unbounded |
| Reported throughput (Mamba’s own benchmark) | Baseline | ~5× same-size Transformer |
That’s a real, structural advantage for anything that streams for a long time on constrained hardware — long documents, long conversations, edge and on-device inference where memory is the scarce resource. It is not, however, the whole story.
Where the Transformer Still Wins: Exact Recall
A fixed-size notebook is a genuine information bottleneck, not just a simplification. Compressing “everything so far” into one constant-size summary means something has to be dropped or blurred — and what gets dropped tends to be exactly the kind of precise, verbatim detail that a full transcript preserves automatically.
Jelassi et al.’s “Repeat After Me” (2024, arXiv:2402.01032) put this to the test directly and found Transformers dramatically outperform SSMs at copying and exact in-context retrieval — tasks that require recalling a specific token verbatim from far back in the input. A Transformer can do this natively: its KV cache keeps every past token individually addressable, so “attend back to word 4,000” is a direct lookup. An SSM’s compressed state has no such direct address book; the detail either survived compression or it didn’t. In practice, that’s the gap between “summarize this long contract” (where an SSM can hold its own) and “quote me the exact clause on page 40” (where the full-attention Transformer has a structural edge).
Why Hybrids Won, Not a “Transformer-Killer”
When Mamba landed in late 2023, plenty of commentary framed it as the beginning of the end for attention. That didn’t happen. Frontier flagship models from the major labs remain Transformer-based, and the recall weakness above is a big part of why a pure-SSM swap-out never stuck.
What actually shipped into production is more interesting than either extreme: hybrids that interleave a small fraction of attention layers into a mostly-SSM stack, borrowing the SSM’s cheap streaming while patching in attention’s precise recall exactly where it’s needed. Jamba, from AI21 and described as the first production-grade Mamba-based model, uses roughly one attention layer for every eight total layers — about 87.5% Mamba — combined with a mixture-of-experts design, reaching a 256K context window and around 3× the throughput of Mixtral on long-context workloads. Zamba, from Zyphra, takes a leaner approach: a Mamba backbone with a single shared attention module reused throughout the network, aiming for near-Transformer quality at a fraction of the parameter cost.
It’s the same broader story LIG covers in its look at the Continuous Thought Machine: researchers actively exploring genuinely non-Transformer or hybrid designs, rather than treating standard attention as the only architecture worth building. Mamba’s real legacy so far isn’t replacing the Transformer — it’s proving that a fundamentally different memory strategy can compete hard enough on its home turf that hybrids became worth building.
❓ Frequently Asked Questions
What is a state-space model (SSM), in plain English? An SSM is an AI architecture that carries forward one fixed-size “compressed summary” of everything it has processed so far, updating it one token at a time, instead of re-comparing every new word against the entire past conversation the way Transformer attention does. That fixed size is why it can stream very long input at constant memory cost.
Is Mamba faster than a Transformer? For training and especially for long-sequence inference, generally yes — the original Mamba paper reports around 5× higher inference throughput than a same-size Transformer, because training cost scales linearly instead of quadratically and inference memory stays flat instead of growing. Real-world gains depend heavily on sequence length and hardware.
Did Mamba replace the Transformer? No. Despite 2023–24 hype calling it a “Transformer-killer,” frontier flagship AI models are still Transformer-based. What actually won adoption is hybrid architectures like Jamba and Zamba, which mix a small amount of attention into a mostly state-space model, rather than either pure architecture on its own.
What’s the catch with Mamba’s fixed-size memory? Compressing an entire conversation into one constant-size state is a real information bottleneck. Research (Jelassi et al., “Repeat After Me”) shows Transformers clearly beat state-space models at copying and exact recall — pulling one precise detail verbatim from far back in the text — because a Transformer’s KV cache keeps every past token individually addressable and an SSM’s compressed state does not.
What is Mamba-2’s “State Space Duality”? It’s a formal proof, introduced in the Mamba-2 paper, that selective state-space models and a form of linear attention are mathematically the same family of computation, just written two different ways. It’s the direct bridge between the “replace attention with a recurrent state” approach and the “reformulate attention’s math to be faster” approach — two research paths that turned out to converge.
📚 Further Reading
- How AI Beats the “n² Wall”: The Attention-Efficiency Playbook — the cornerstone guide to every technique in this hub, including where state-space models fit.
- What Is Linear Attention? — the kernel-trick cousin that Mamba-2 formally proved is the same math family as SSMs.
- From Indexing to Internal Thought: How Continuous Thought Machines May Reinvent Search — another architecture exploring what AI can do beyond vanilla attention.
- MQA vs. GQA vs. MLA: Three Ways AI Models Shrink Their Memory — how Transformers themselves tackle the KV-cache problem SSMs sidestep entirely.
📚 Recommended Resources
Disclosure: some links below are Amazon affiliate links. If you buy through them, I earn a small commission — at no extra cost to you — which helps keep this site running.
Artificial Intelligence: A Guide for Thinking Humans
Melanie Mitchell · Book · ~$18 on Amazon
Before you decide whether Mamba is a “Transformer-killer” or just a clever trade-off, it’s worth building the de-hyped, conceptual grounding to weigh claims like that for yourself. Mitchell’s book is the clearest antidote to architecture hype we’ve found — exactly the mindset this comparison tries to model.
Build a Large Language Model (From Scratch)
Sebastian Raschka · Book · ~$45 on Amazon
If reading about a “fixed-size hidden state” versus a “growing KV cache” made you want to see the actual mechanics, this is the book that gets you there — building attention and its alternatives yourself, line by line, instead of taking any benchmark on faith.
Natural Language Processing with Transformers
Lewis Tunstall, Leandro von Werra, Thomas Wolf · Book · ~$50 on Amazon
The Transformer side of this comparison — the architecture that still wins on exact recall — is covered in practical depth here, from Hugging Face engineers who work with it daily. A useful counterweight if you want the mainstream approach’s own strongest case.