Definition
State-space models (SSMs) are sequence models whose compute grows linearly with the length of the input, in contrast to the quadratic growth of the attention mechanism that powers transformers. Mamba, introduced by Albert Gu and Tri Dao in December 2023, is the state-space design that first made this class competitive with transformers on language, and it is the reason the term is worth knowing.
The mechanism a reader should walk away with: instead of comparing every token with every other token, an SSM carries a small, fixed-size state — think of it as a running summary — and updates that state once per token as it reads the sequence left to right. The past is not re-examined; it is compressed into the state. That single design choice is what buys the linear scaling, and it is also what makes SSMs harder to get right, because a fixed-size summary has to decide what to throw away.
If you are here to know whether Mamba has replaced the transformer: it has not, as of 2026. It has carved out real ground — long-context models, cheap inference, and non-language domains like genomics — and the frontier has mostly absorbed it as hybrids that mix a few attention layers among many state-space layers, rather than as a wholesale swap.
How It Works
An SSM keeps a hidden state vector h. At each step it reads the next token's representation x, updates the state with a linear rule, and produces an output y:
h_t = A · h_ + B · x_t (carry the state forward, mix in the new token) y_t = C · h_t (read the output off the state)
That is the whole loop. A controls how much of the old state survives, B how the new token is written in, and C how the state is read out. Crucially, every operation here is linear — there is no nonlinearity wrapped around the recurrence. That linearity is the pivot on which everything else turns, so it is worth being precise about two consequences.
The recurrent-and-convolutional dual view
Because the update is linear, you can unroll the loop and see that each output is just a weighted sum of all past inputs, where the weights are powers of A. Written that way, the entire sequence can be computed as a single convolution — one big parallel operation over all positions at once, which is exactly what a GPU is built to do fast. But the same model can also be run as the step-by-step recurrence above, which needs only the current token and the fixed-size state — no growing cache, constant memory per step.
This is the dual view that makes SSMs special: train as a convolution (parallel, fast), run as a recurrence (constant memory, cheap per token). A classic RNN cannot do the first, because its recurrence is nonlinear and every step genuinely depends on the finished result of the previous one.
Why linear beats quadratic — worked
The payoff is in how the two families scale. Attention computes a score for every pair of positions, so for a sequence of length n it does work proportional to n². An SSM does one constant-size state update per token, so its work is proportional to n. Those two curves diverge fast:
- Double the sequence length and attention's cost quadruples (2² = 4); the SSM's merely doubles.
- Go from a 1,000-token prompt to a 64,000-token one — 64× longer — and attention does 64² = 4,096× more pairwise work, while the SSM does just 64× more.
The gap between them is not a constant; it is itself proportional to n. At 256,000 tokens the ratio in raw sequence-mixing operations is on the order of 256,000 to 1. That is the durable fact behind every claim about SSMs and long context. (The ## Code Example below prints this table so you can see the two columns pull apart.)
The change that mattered: selection (S4 to S6)
Earlier state-space models — the S4 line that preceded Mamba — used fixed A, B, and C matrices: the state was updated the same way regardless of what the model was reading. That made the convolution trick clean, but it also meant the model could not choose to pay attention to an important token and ignore filler, which is why S4 lagged transformers on language.
Mamba's contribution is to make those parameters functions of the input token — the paper's phrasing is "letting the SSM parameters be functions of the input... allowing the model to selectively propagate or forget information." The authors call this the selective mechanism and abbreviate the resulting layer S6, because, in their words, it comprises "S4 models with a selection mechanism and computed with a scan."
There is a cost: once A and B depend on the token, the model is no longer time-invariant, so the neat convolution shortcut is gone. Mamba recovers parallel training with a hardware-aware parallel scan — a GPU kernel that computes the recurrence across positions without ever materializing the full state history in slow memory. This is the combination — selection for quality, scan for speed — that let Mamba report, in its abstract, "5× higher throughput than Transformers and linear scaling in sequence length," with a "Mamba-3B model [that] outperforms Transformers of the same size and matches Transformers twice its size."
Real-World Applications
State-space layers have shipped in named production systems, and the shape of where they show up tells you exactly what they are good for.
Hybrid language models. AI21 Labs' Jamba interleaves Transformer and Mamba blocks (roughly one attention layer for every several state-space layers) with a mixture-of-experts, and ships an effective context window of 256,000 tokens — a length that is expensive to serve with pure attention because of the growing key-value cache. This hybrid pattern, not a pure-Mamba stack, is the dominant way SSMs reach the frontier: the state-space layers carry the bulk of the sequence cheaply, and the few attention layers restore the exact-recall ability the state-space layers lack.
Code models. Mistral AI's Codestral Mamba (a Mamba-2-based 7B code model) leans on the linear-time inference property: because cost per token does not grow with how much code is already in context, it stays responsive on very long files where an attention model slows down.
Pure-SSM open models. TII's Falcon Mamba 7B shipped as an attention-free state-space language model, a proof that the architecture can stand entirely on its own for general language, not only inside hybrids.
Non-language sequences. The original Mamba paper reports results on DNA sequences and audio waveforms — domains with sequences far longer than a typical text prompt, where quadratic attention is simply impractical and the linear scaling is not a convenience but the enabling condition. Genomics work has been an especially active adopter for exactly this reason.
Key Concepts
- State (the running summary): a fixed-size vector that holds everything the model has decided to remember. Its size is a capacity ceiling — unlike attention, which can reach back to any exact token, an SSM has already compressed the past and cannot recover a detail it chose to discard.
- Linear recurrence: the update
h_t = A·h_{t-1} + B·x_thas no nonlinearity around it. This is the single property that separates an SSM from an RNN and unlocks parallel training. - Selection (S6): making the update depend on the current input, so the model can gate what enters and leaves the state. This is the S4-to-Mamba change.
- Parallel scan: the GPU kernel that computes a time-varying linear recurrence across all positions at once, replacing the convolution that selection made unavailable.
Challenges
The honest limitation is recall. Because the state is a fixed-size summary, a pure state-space model is measurably worse than attention at tasks that require reproducing a specific earlier token verbatim — copying a long string, retrieving an exact value from far back in the context, or in-context "look up the token that followed this pattern last time" behavior. Attention can always point straight at the original token; an SSM can only consult whatever survived compression. This is not a tuning problem, it is the direct consequence of trading a growing cache for a constant-size state, and it is the specific weakness hybrids exist to patch.
A second challenge is maturity and tooling. The transformer has a decade of kernels, quantization tricks, serving stacks, and accumulated engineering behind it; state-space layers are newer, so much of that ecosystem has had to be rebuilt, and a lab choosing an architecture weighs that friction, not just the asymptotics.
Third, the asymptotic win is not the whole story at short lengths. Quadratic cost only dominates once sequences are long; for short prompts the constant factors and the highly optimized state of attention kernels can leave a transformer competitive or faster. The linear-scaling argument is an argument about long sequences specifically.
Future Trends
As of mid-2026, the settled direction is hybrids, not replacement — most new long-context designs interleave a minority of attention layers with a majority of state-space (or other sub-quadratic) layers, keeping attention's exact recall where it is cheap while paying linear cost for the bulk of the sequence. Expect continued work on the recall gap directly (larger or structured states, better selection), on the serving side (state-space models change what an inference stack caches, which interacts with the memory-bound nature of decoding), and on non-language domains where sequence lengths make quadratic attention a non-starter. The architectural monoculture around pure attention is the thing that has genuinely ended; a single successor has not been crowned.
Code Example
The linear-versus-quadratic gap is the whole argument, so it is worth seeing on real numbers. This prints, for growing sequence lengths, the sequence-mixing work done by attention (proportional to n²) against a state-space model (proportional to n), and their ratio.
lengths = [1_000, 4_000, 16_000, 64_000, 256_000]
print(f"{'seq len':>9} | {'attention (n^2)':>16} | {'SSM (n)':>10} | {'attn/SSM':>9}")
print("-" * 54)
for n in lengths:
attn = n * n # pairwise scores: quadratic in length
ssm = n # one constant-work state update per token: linear
print(f"{n:>9,} | {attn:>16,} | {ssm:>10,} | {attn // ssm:>7,}x")
Output:
seq len | attention (n^2) | SSM (n) | attn/SSM
------------------------------------------------------
1,000 | 1,000,000 | 1,000 | 1,000x
4,000 | 16,000,000 | 4,000 | 4,000x
16,000 | 256,000,000 | 16,000 | 16,000x
64,000 | 4,096,000,000 | 64,000 | 64,000x
256,000 | 65,536,000,000 | 256,000 | 256,000x
The last column is the sequence length — because n² / n = n, the factor by which attention outspends a state-space model on sequence mixing grows in lock-step with how long the sequence gets. That single line is why long-context and streaming systems reach for this architecture, and why the reach for it grows with the context window.
A note on recurrence, reconciled
Two pages in this glossary can look like they disagree. The recurrent neural network page says recurrence "lost the language crown" and now "lives mainly in streaming and resource-constrained corners rather than at the frontier." The attention mechanism page says state-space models "abandon the n² matrix entirely," which is "why nearly every long-context design is a hybrid rather than a replacement." Both are correct, because they are talking about two different things wearing the same word.
What lost is the nonlinear recurrence of the classic RNN: a loop that must run strictly step by step, cannot be parallelized across time, and drags a vanishing-gradient problem behind it. That did not come back and is not at the frontier. What returned is the linear recurrence of the state-space model — mathematically a loop, but because it is linear it can be unrolled into a parallel operation for training, then run as a constant-memory loop for inference. It recovers the one thing recurrence was always good at (cheap, streaming, fixed-memory processing) without the training penalty that sank the RNN, and Mamba's selection mechanism closed enough of the quality gap to make it competitive on language. So recurrence did not "come back" so much as it was rebuilt into a form the RNN never had — which is exactly why the frontier answer is hybrids and specific niches, not a wholesale replacement of attention.