{"course":{"slug":"how-llms-work","title":"How a Language Model Actually Works","url":"https://addaly.com/learn/how-llms-work","level":"intermediate","track":"start"},"lesson":{"slug":"beyond-the-transformer","title":"Beyond the transformer: what the alternatives trade","url":"https://addaly.com/learn/how-llms-work/beyond-the-transformer","position":68,"of":78,"minutes":9,"answer":"Every alternative to attention replaces an exact, growing cache with a fixed-size summary, buying constant per-token cost at the price of exact long-range recall, and none of them changes the failure modes.","keyIdea":"Every alternative to attention replaces an exact, growing cache with a fixed-size summary, buying constant per-token cost at the price of exact long-range recall, and none of them changes the failure modes, because those come from the training setup rather than the attention layer."},"body":{"format":"markdown","text":"## Why anyone wants out\n\nRead back over this module and one object keeps appearing: the key-value cache. It grows with every token of context, it is read on every decode step, it decides how many users fit on a card, and prefill cost grows with the square of the prompt. Paging, quantising the cache, sliding windows, grouped-query attention, disaggregated serving: most of the engineering in the last nine lessons is engineering around one data structure.\n\nThe alternatives to the transformer attack that structure at the architecture rather than in the serving stack. Each one makes a trade, and the trade is the thing to understand, because the marketing will only tell you one side of it.\n\n## State-space models\n\nThe idea, in its 2023 form, replaces attention with a **fixed-size state** (a block of numbers of constant size) that is updated once per token. The update rule depends on the input, which is what makes the model selective about what it keeps. Per-token cost is constant no matter how long the context; there is no cache that grows; decode reads the weights plus a state that fits in a few megabytes. Training remains parallel, through a scan operation rather than through attention's all-pairs matrix.\n\nThe trade follows directly from the words \"fixed size\". Attention's cache is exact: every earlier token's key and value is kept, and the model can look any of them up. A fixed-size state must **compress** the whole history into a constant number of numbers, and compression is lossy. On tasks that require pulling an arbitrary detail from far back (a needle in a long document, copying a long identifier, learning from many in-context examples) pure state-space models measure weaker than transformers of the same size. They remember the gist and lose the specifics. Attention has exact memory; a state has good memory.\n\n## Linear attention and its relatives\n\nAn older idea, revived: rewrite attention so that the softmax is replaced by a function that lets the running sum be maintained as a single matrix updated per token. The per-token cost becomes constant and the cache disappears, at the same price: a fixed-size summary in place of exact lookup. Recent versions add gating, so the model can forget selectively, and the strongest of them are close to state-space models in behaviour. Several open models of this family exist and are worth knowing about mainly because they run well on ordinary hardware.\n\n## Hybrids: where the field actually landed\n\nMost production use of these ideas in 2024 and 2025 is neither pure. A hybrid interleaves many state-space or linear layers with a few attention layers: one in eight is a common ratio. The attention layers restore exact recall; the state-space layers cut the cache by roughly the ratio and make most of the per-token cost constant. Module two's sliding-window hybrids are the same instinct approached from the transformer side.\n\nThis is worth stating plainly because the argument is often framed as a replacement. It has not been a replacement. It has been a mix, and the mix is winning in cost-sensitive, long-context serving while pure transformers remain dominant at the frontier.\n\n## Diffusion language models\n\nA different escape. Instead of generating left to right, start with a block of noise tokens and refine all of them in parallel over several passes, the way image diffusion models denoise a picture. The first commercial ones appeared in 2025 and they are fast, because each pass touches every position at once rather than one.\n\nThe trades are less settled. Output comes in fixed-length blocks. Quality on long, coherent text lags the best autoregressive models. And module three's mechanism (that a longer answer is more computation, one token of thinking at a time) works differently when all tokens are produced together; how reasoning behaves in such a model is an open question. Short completions and code are where the early results look strongest.\n\n## What does not change\n\nNow the part that this course has earned. Tokens, embeddings, the next-token objective, the context as one undifferentiated sequence, preference training: none of these is a property of the attention layer. They are properties of the training setup. So a state-space model hallucinates by exactly the mechanism of module six's first lesson. It is prompt-injectable. It is sycophantic. It drifts between versions. The failure modes of module six survive the architecture change untouched. What moves is the cost curve, and only the cost curve.\n\n## How to read a new-architecture claim\n\nFour questions.\n\n1. **At what context length does the claimed speed-up appear?** Constant per-token cost beats attention only once the context is long enough for the cache to matter: often past several thousand tokens. At a thousand tokens there is nothing to win.\n2. **What is the exact-recall score against a transformer of the same size?** A needle-in-a-haystack or long-copy result. If it is absent from the announcement, that is the trade being hidden.\n3. **Is it a hybrid?** If so, how many attention layers, because that number is the cache saving and the recall floor at once.\n4. **Was the training recipe comparable?** Many comparisons pit a new architecture on fresh data against an old transformer on old data, and the data is doing the work.\n\n## The honest state\n\nAt the top end, transformers hold. In the middle, hybrids are taking the long-context, cost-sensitive deployments. Pure state-space and linear models are strong at small scale and unproven at the largest, and diffusion models are early. The safe prediction is more hybrid rather than a winner, and (the thread this module has followed throughout) that whichever design serves your next model, the size of its cache will still be the number that decides what it costs."},"check":{"question":"A hybrid model with one attention layer for every eight state-space layers advertises a key-value cache roughly eight times smaller than a transformer of the same size. What capability was traded to get it?","options":[{"text":"Training stability, since state-space layers are harder to keep in a workable numerical range across many layers.","correct":false,"tempting":"Normalisation and stability are real concerns in deep models, but they are not what the fixed-size state costs; the trade is in what the model can retrieve, not in whether it trains."},{"text":"Nothing was traded; the state-space layers are strictly better and the remaining attention layers are kept only for compatibility with existing serving engines.","correct":false,"tempting":"If that were true the hybrid would have zero attention layers; the ones kept are there precisely because pure state-space models lose something measurable."},{"text":"The ability to process a prompt in parallel, because recurrent state-space layers must consume the prompt's tokens strictly one at a time during prefill.","correct":false,"tempting":"Sounds like a necessary consequence of recurrence, but these layers are trained and prefilled with a parallel scan; the sequential cost applies to decode, where it is a feature."},{"text":"Exact recall of arbitrary earlier tokens, which a fixed-size state cannot hold.","correct":true}]},"prev":{"slug":"cascades-and-routing","title":"Cascades and routers: paying for the hard cases only","url":"https://addaly.com/learn/how-llms-work/cascades-and-routing"},"next":{"slug":"superposition-and-features","title":"Superposition: more features than dimensions","url":"https://addaly.com/learn/how-llms-work/superposition-and-features"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Beyond the transformer: what the alternatives trade', in How a Language Model Actually Works. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/how-llms-work/beyond-the-transformer (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Beyond the transformer: what the alternatives trade","courseTitle":"How a Language Model Actually Works","module":{"index":7,"slug":"running-it","title":"Running it: memory, speed and money"},"answer":null,"keyIdea":"Every alternative to attention replaces an exact, growing cache with a fixed-size summary, buying constant per-token cost at the price of exact long-range recall, and none of them changes the failure modes, because those come from the training setup rather than the attention layer.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"why-anyone-wants-out","runs":[{"text":"Why anyone wants out"}]},{"t":"p","runs":[{"text":"Read back over this module and one object keeps appearing: the key-value cache. It grows with every token of context, it is read on every decode step, it decides how many users fit on a card, and prefill cost grows with the square of the prompt. Paging, quantising the cache, sliding windows, grouped-query attention, disaggregated serving: most of the engineering in the last nine lessons is engineering around one data structure."}]},{"t":"p","runs":[{"text":"The alternatives to the transformer attack that structure at the architecture rather than in the serving stack. Each one makes a trade, and the trade is the thing to understand, because the marketing will only tell you one side of it."}]},{"t":"h2","id":"state-space-models","runs":[{"text":"State-space models"}]},{"t":"p","runs":[{"text":"The idea, in its 2023 form, replaces attention with a "},{"text":"fixed-size state","b":true},{"text":" (a block of numbers of constant size) that is updated once per token. The update rule depends on the input, which is what makes the model selective about what it keeps. Per-token cost is constant no matter how long the context; there is no cache that grows; decode reads the weights plus a state that fits in a few megabytes. Training remains parallel, through a scan operation rather than through attention’s all-pairs matrix."}]},{"t":"p","runs":[{"text":"The trade follows directly from the words “fixed size”. Attention’s cache is exact: every earlier token’s key and value is kept, and the model can look any of them up. A fixed-size state must "},{"text":"compress","b":true},{"text":" the whole history into a constant number of numbers, and compression is lossy. On tasks that require pulling an arbitrary detail from far back (a needle in a long document, copying a long identifier, learning from many in-context examples) pure state-space models measure weaker than transformers of the same size. They remember the gist and lose the specifics. Attention has exact memory; a state has good memory."}]},{"t":"h2","id":"linear-attention-and-its-relatives","runs":[{"text":"Linear attention and its relatives"}]},{"t":"p","runs":[{"text":"An older idea, revived: rewrite attention so that the softmax is replaced by a function that lets the running sum be maintained as a single matrix updated per token. The per-token cost becomes constant and the cache disappears, at the same price: a fixed-size summary in place of exact lookup. Recent versions add gating, so the model can forget selectively, and the strongest of them are close to state-space models in behaviour. Several open models of this family exist and are worth knowing about mainly because they run well on ordinary hardware."}]},{"t":"h2","id":"hybrids-where-the-field-actually-landed","runs":[{"text":"Hybrids: where the field actually landed"}]},{"t":"p","runs":[{"text":"Most production use of these ideas in 2024 and 2025 is neither pure. A hybrid interleaves many state-space or linear layers with a few attention layers: one in eight is a common ratio. The attention layers restore exact recall; the state-space layers cut the cache by roughly the ratio and make most of the per-token cost constant. Module two’s sliding-window hybrids are the same instinct approached from the transformer side."}]},{"t":"p","runs":[{"text":"This is worth stating plainly because the argument is often framed as a replacement. It has not been a replacement. It has been a mix, and the mix is winning in cost-sensitive, long-context serving while pure transformers remain dominant at the frontier."}]},{"t":"h2","id":"diffusion-language-models","runs":[{"text":"Diffusion language models"}]},{"t":"p","runs":[{"text":"A different escape. Instead of generating left to right, start with a block of noise tokens and refine all of them in parallel over several passes, the way image diffusion models denoise a picture. The first commercial ones appeared in 2025 and they are fast, because each pass touches every position at once rather than one."}]},{"t":"p","runs":[{"text":"The trades are less settled. Output comes in fixed-length blocks. Quality on long, coherent text lags the best autoregressive models. And module three’s mechanism (that a longer answer is more computation, one token of thinking at a time) works differently when all tokens are produced together; how reasoning behaves in such a model is an open question. Short completions and code are where the early results look strongest."}]},{"t":"h2","id":"what-does-not-change","runs":[{"text":"What does not change"}]},{"t":"p","runs":[{"text":"Now the part that this course has earned. Tokens, embeddings, the next-token objective, the context as one undifferentiated sequence, preference training: none of these is a property of the attention layer. They are properties of the training setup. So a state-space model hallucinates by exactly the mechanism of module six’s first lesson. It is prompt-injectable. It is sycophantic. It drifts between versions. The failure modes of module six survive the architecture change untouched. What moves is the cost curve, and only the cost curve."}]},{"t":"h2","id":"how-to-read-a-new-architecture-claim","runs":[{"text":"How to read a new-architecture claim"}]},{"t":"p","runs":[{"text":"Four questions."}]},{"t":"ol","start":1,"items":[[{"text":"At what context length does the claimed speed-up appear?","b":true},{"text":" Constant per-token cost beats attention only once the context is long enough for the cache to matter: often past several thousand tokens. At a thousand tokens there is nothing to win."}],[{"text":"What is the exact-recall score against a transformer of the same size?","b":true},{"text":" A needle-in-a-haystack or long-copy result. If it is absent from the announcement, that is the trade being hidden."}],[{"text":"Is it a hybrid?","b":true},{"text":" If so, how many attention layers, because that number is the cache saving and the recall floor at once."}],[{"text":"Was the training recipe comparable?","b":true},{"text":" Many comparisons pit a new architecture on fresh data against an old transformer on old data, and the data is doing the work."}]]},{"t":"h2","id":"the-honest-state","runs":[{"text":"The honest state"}]},{"t":"p","runs":[{"text":"At the top end, transformers hold. In the middle, hybrids are taking the long-context, cost-sensitive deployments. Pure state-space and linear models are strong at small scale and unproven at the largest, and diffusion models are early. The safe prediction is more hybrid rather than a winner, and (the thread this module has followed throughout) that whichever design serves your next model, the size of its cache will still be the number that decides what it costs."}]}],"check":{"question":[{"text":"A hybrid model with one attention layer for every eight state-space layers advertises a key-value cache roughly eight times smaller than a transformer of the same size. What capability was traded to get it?"}],"options":[{"text":[{"text":"Training stability, since state-space layers are harder to keep in a workable numerical range across many layers."}],"correct":false,"tempting":[{"text":"Normalisation and stability are real concerns in deep models, but they are not what the fixed-size state costs; the trade is in what the model can retrieve, not in whether it trains."}]},{"text":[{"text":"Nothing was traded; the state-space layers are strictly better and the remaining attention layers are kept only for compatibility with existing serving engines."}],"correct":false,"tempting":[{"text":"If that were true the hybrid would have zero attention layers; the ones kept are there precisely because pure state-space models lose something measurable."}]},{"text":[{"text":"The ability to process a prompt in parallel, because recurrent state-space layers must consume the prompt’s tokens strictly one at a time during prefill."}],"correct":false,"tempting":[{"text":"Sounds like a necessary consequence of recurrence, but these layers are trained and prefilled with a parallel scan; the sequential cost applies to decode, where it is a feature."}]},{"text":[{"text":"Exact recall of arbitrary earlier tokens, which a fixed-size state cannot hold."}],"correct":true,"tempting":null}],"order":[1,2,3,0],"why":null},"prev":{"slug":"cascades-and-routing","title":"Cascades and routers: paying for the hard cases only","minutes":8},"next":{"slug":"superposition-and-features","title":"Superposition: more features than dimensions","minutes":9}}}