Beyond the transformer: what the alternatives trade

Why anyone wants out

Read back over this module and one object keeps appearing: the key-value cache. It grows with every token of context, it is read on every decode step, it decides how many users fit on a card, and prefill cost grows with the square of the prompt. Paging, quantising the cache, sliding windows, grouped-query attention, disaggregated serving: most of the engineering in the last nine lessons is engineering around one data structure.

The alternatives to the transformer attack that structure at the architecture rather than in the serving stack. Each one makes a trade, and the trade is the thing to understand, because the marketing will only tell you one side of it.

State-space models

The idea, in its 2023 form, replaces attention with a fixed-size state (a block of numbers of constant size) that is updated once per token. The update rule depends on the input, which is what makes the model selective about what it keeps. Per-token cost is constant no matter how long the context; there is no cache that grows; decode reads the weights plus a state that fits in a few megabytes. Training remains parallel, through a scan operation rather than through attention’s all-pairs matrix.

The trade follows directly from the words “fixed size”. Attention’s cache is exact: every earlier token’s key and value is kept, and the model can look any of them up. A fixed-size state must compress the whole history into a constant number of numbers, and compression is lossy. On tasks that require pulling an arbitrary detail from far back (a needle in a long document, copying a long identifier, learning from many in-context examples) pure state-space models measure weaker than transformers of the same size. They remember the gist and lose the specifics. Attention has exact memory; a state has good memory.

Linear attention and its relatives

An older idea, revived: rewrite attention so that the softmax is replaced by a function that lets the running sum be maintained as a single matrix updated per token. The per-token cost becomes constant and the cache disappears, at the same price: a fixed-size summary in place of exact lookup. Recent versions add gating, so the model can forget selectively, and the strongest of them are close to state-space models in behaviour. Several open models of this family exist and are worth knowing about mainly because they run well on ordinary hardware.

Hybrids: where the field actually landed

Most production use of these ideas in 2024 and 2025 is neither pure. A hybrid interleaves many state-space or linear layers with a few attention layers: one in eight is a common ratio. The attention layers restore exact recall; the state-space layers cut the cache by roughly the ratio and make most of the per-token cost constant. Module two’s sliding-window hybrids are the same instinct approached from the transformer side.

This is worth stating plainly because the argument is often framed as a replacement. It has not been a replacement. It has been a mix, and the mix is winning in cost-sensitive, long-context serving while pure transformers remain dominant at the frontier.

Diffusion language models

A different escape. Instead of generating left to right, start with a block of noise tokens and refine all of them in parallel over several passes, the way image diffusion models denoise a picture. The first commercial ones appeared in 2025 and they are fast, because each pass touches every position at once rather than one.

The trades are less settled. Output comes in fixed-length blocks. Quality on long, coherent text lags the best autoregressive models. And module three’s mechanism (that a longer answer is more computation, one token of thinking at a time) works differently when all tokens are produced together; how reasoning behaves in such a model is an open question. Short completions and code are where the early results look strongest.

What does not change

Now the part that this course has earned. Tokens, embeddings, the next-token objective, the context as one undifferentiated sequence, preference training: none of these is a property of the attention layer. They are properties of the training setup. So a state-space model hallucinates by exactly the mechanism of module six’s first lesson. It is prompt-injectable. It is sycophantic. It drifts between versions. The failure modes of module six survive the architecture change untouched. What moves is the cost curve, and only the cost curve.

How to read a new-architecture claim

Four questions.

1. At what context length does the claimed speed-up appear? Constant per-token cost beats attention only once the context is long enough for the cache to matter: often past several thousand tokens. At a thousand tokens there is nothing to win.
2. What is the exact-recall score against a transformer of the same size? A needle-in-a-haystack or long-copy result. If it is absent from the announcement, that is the trade being hidden.
3. Is it a hybrid? If so, how many attention layers, because that number is the cache saving and the recall floor at once.
4. Was the training recipe comparable? Many comparisons pit a new architecture on fresh data against an old transformer on old data, and the data is doing the work.

The honest state

At the top end, transformers hold. In the middle, hybrids are taking the long-context, cost-sensitive deployments. Pure state-space and linear models are strong at small scale and unproven at the largest, and diffusion models are early. The safe prediction is more hybrid rather than a winner, and (the thread this module has followed throughout) that whichever design serves your next model, the size of its cache will still be the number that decides what it costs.