{"course":{"slug":"how-llms-work","title":"How a Language Model Actually Works","url":"https://addaly.com/learn/how-llms-work","level":"intermediate","track":"start"},"lesson":{"slug":"sparse-autoencoders","title":"Sparse autoencoders: a dictionary for the residual stream","url":"https://addaly.com/learn/how-llms-work/sparse-autoencoders","position":71,"of":78,"minutes":10,"answer":"A wide autoencoder trained to reconstruct activations under a sparsity penalty pulls superposed directions apart into thousands of single-meaning features without labels: a real advance in discovering what a model represents.","keyIdea":"A wide autoencoder trained to reconstruct activations under a sparsity penalty pulls superposed directions apart into thousands of single-meaning features without labels: a real advance in discovering what a model represents, and not yet a reliable instrument for auditing it."},"body":{"format":"markdown","text":"## The bet\n\nProbes find the directions you can name. Superposition says there are thousands you cannot. The sparse autoencoder is a bet that they can be found anyway, without labels, from two assumptions the first lesson of this module set up: features are directions, and features are sparse.\n\n## The mechanism\n\nTake activations from one layer of the model: say the 4,096-dimensional residual stream. Train a small network to reconstruct them through a hidden layer that is much **wider** than the input: 65,000 dimensions, or a million.\n\n```text\nh      = relu( W_enc · x + b_enc )      # 4,096  -> 65,536, sparse\nx_hat  = W_dec · h + b_dec              # 65,536 -> 4,096\nloss   = |x - x_hat|²  +  λ · |h|₁\n```\n\nThe first term wants a faithful reconstruction. The second penalises the hidden layer for having many non-zero entries. Together they say: explain each activation as a sum of as *few* dictionary entries as possible. Each column of the decoder is a candidate feature direction, and the sparsity is what pulls the overlapping directions of superposition apart into separate columns: without it, the autoencoder would learn the identity and tell you nothing. Some versions replace the penalty with \"keep only the top *k* entries\", which does the same job more controllably.\n\nTraining data is tens of millions of activations from real text. For a small model this is a few hours on one GPU; for a production-scale model, days on many.\n\n## What was found\n\nOn a one-layer model in 2023, the dictionary contained thousands of features that were clean in a way neurons never were: Arabic script, DNA sequences, HTTP request headers, legal boilerplate, each firing on exactly what its name suggests.\n\nApplied to production-scale models in 2024, the dictionaries reached millions of entries and the features became abstract. Code containing a bug. Sycophantic praise. A specific bridge. A specific person. Deception in a described scenario. Features that fire for the same concept across languages, and across text and images in a multimodal model: the Golden Gate Bridge feature fires on the opening sentence of the bridge's Wikipedia article in several languages, and on photographs of the bridge, although the dictionary was trained only on text.\n\nThe demonstration that made the idea public was a model with one feature amplified (the bridge) that then brought the bridge into every answer, including its account of its own identity. Behaviour followed the direction, which is the steering lesson to come.\n\nTwo free resources make all of this something you can do rather than read about. Gemma Scope publishes trained dictionaries for every layer of an open 2-billion-parameter model. Neuronpedia lets you browse the features in a browser, see what each fires on, and run your own text through them. Loading a dictionary and listing the features active on a sentence is twenty lines with the `sae_lens` library in a free notebook.\n\n## Reading a feature\n\nA feature is described three ways. Its **top-activating examples** (the passages where it fires hardest) give it a name. Its **logit effect** (which tokens become more likely when its direction is added) says what it does to the output. And **steering** (adding or removing it and watching the behaviour) says whether the name and the effect are real.\n\nOne property surprises everyone. Train a bigger dictionary on the same model and a single \"birds\" feature splits into species. Bigger again and the species split into contexts: a bird in a poem, a bird in a field guide. Features are not a fixed inventory the model has; they are a decomposition at a resolution you chose. The dictionary size is a dial, not a discovery.\n\n## The limits, which are the important part\n\n**Reconstruction is imperfect.** Replace the model's activations with the dictionary's reconstruction and run the model: the loss rises measurably. Some fraction of the computation is not captured by any feature (the dictionary's dark matter) and nobody can yet say what it does.\n\n**Many features resist interpretation**, and some dictionary entries are dead, never firing, or dense, firing on everything. The count of clean features is impressive and it is a fraction of the dictionary.\n\n**Legibility is not mechanism.** The dictionary is a basis chosen for our convenience. A feature that reads clearly to a person may not be a unit the model computes with; it may be a projection of several such units that happen to align with a human concept.\n\n**On downstream tasks the record is mixed.** When dictionary features have been compared with plain linear probes on detection tasks (does this prompt contain a harmful request, is this output in the wrong language) probes have often matched or beaten them. Sparse autoencoders are a genuine advance in *discovering* what a model represents. They are not yet a reliable instrument for *auditing* it, and claims that a model has been \"checked\" with them should be read with that distinction in mind.\n\n## What you can do with one this week\n\nRun your own prompts through a published dictionary and list the features that fire; it is the fastest way to see how a model carves up your domain. Use a feature as a cheap classifier in the forward pass. Check whether a \"harmful request\" feature fires on the jailbreak examples of module six, and whether the refusal that follows tracks it. None of that needs more than a free notebook, and it turns the previous four lessons into something you have seen with your own eyes."},"check":{"question":"In training a sparse autoencoder on a model's activations, what does the sparsity penalty do that plain reconstruction would not?","options":[{"text":"It shortens training, because the autoencoder can skip updating dictionary entries that are currently zero.","correct":false,"tempting":"Sparsity does allow some computational shortcuts, but that is a side effect of implementation, not the reason the penalty is there."},{"text":"It filters noise out of the activations, so the reconstruction is closer to the model's true signal than the raw activations were.","correct":false,"tempting":"Backwards: the penalty makes reconstruction worse, not better, and the gap it opens is the dictionary's dark matter, not removed noise."},{"text":"It forces the hidden layer to reproduce the model's own neurons one-to-one, giving the dictionary a stable basis that can be compared across models.","correct":false,"tempting":"The dictionary is wider than the neuron basis precisely so as not to reproduce it, and its features do not transfer across models without retraining."},{"text":"It makes each input explainable by few dictionary entries, which pulls overlapping directions apart into separate features.","correct":true}]},"prev":{"slug":"probing-what-is-represented","title":"Probing: asking the activations a question","url":"https://addaly.com/learn/how-llms-work/probing-what-is-represented"},"next":{"slug":"activation-patching-and-circuits","title":"Patching: finding which parts do the work","url":"https://addaly.com/learn/how-llms-work/activation-patching-and-circuits"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Sparse autoencoders: a dictionary for the residual stream', in How a Language Model Actually Works. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/how-llms-work/sparse-autoencoders (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Sparse autoencoders: a dictionary for the residual stream","courseTitle":"How a Language Model Actually Works","module":{"index":8,"slug":"looking-inside","title":"Looking inside: what the weights actually hold"},"answer":null,"keyIdea":"A wide autoencoder trained to reconstruct activations under a sparsity penalty pulls superposed directions apart into thousands of single-meaning features without labels: a real advance in discovering what a model represents, and not yet a reliable instrument for auditing it.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"the-bet","runs":[{"text":"The bet"}]},{"t":"p","runs":[{"text":"Probes find the directions you can name. Superposition says there are thousands you cannot. The sparse autoencoder is a bet that they can be found anyway, without labels, from two assumptions the first lesson of this module set up: features are directions, and features are sparse."}]},{"t":"h2","id":"the-mechanism","runs":[{"text":"The mechanism"}]},{"t":"p","runs":[{"text":"Take activations from one layer of the model: say the 4,096-dimensional residual stream. Train a small network to reconstruct them through a hidden layer that is much "},{"text":"wider","b":true},{"text":" than the input: 65,000 dimensions, or a million."}]},{"t":"code","lang":"text","label":"Text","text":"h      = relu( W_enc · x + b_enc )      # 4,096  -> 65,536, sparse\nx_hat  = W_dec · h + b_dec              # 65,536 -> 4,096\nloss   = |x - x_hat|²  +  λ · |h|₁"},{"t":"p","runs":[{"text":"The first term wants a faithful reconstruction. The second penalises the hidden layer for having many non-zero entries. Together they say: explain each activation as a sum of as "},{"text":"few","i":true},{"text":" dictionary entries as possible. Each column of the decoder is a candidate feature direction, and the sparsity is what pulls the overlapping directions of superposition apart into separate columns: without it, the autoencoder would learn the identity and tell you nothing. Some versions replace the penalty with “keep only the top "},{"text":"k","i":true},{"text":" entries”, which does the same job more controllably."}]},{"t":"p","runs":[{"text":"Training data is tens of millions of activations from real text. For a small model this is a few hours on one GPU; for a production-scale model, days on many."}]},{"t":"h2","id":"what-was-found","runs":[{"text":"What was found"}]},{"t":"p","runs":[{"text":"On a one-layer model in 2023, the dictionary contained thousands of features that were clean in a way neurons never were: Arabic script, DNA sequences, HTTP request headers, legal boilerplate, each firing on exactly what its name suggests."}]},{"t":"p","runs":[{"text":"Applied to production-scale models in 2024, the dictionaries reached millions of entries and the features became abstract. Code containing a bug. Sycophantic praise. A specific bridge. A specific person. Deception in a described scenario. Features that fire for the same concept across languages, and across text and images in a multimodal model: the Golden Gate Bridge feature fires on the opening sentence of the bridge’s Wikipedia article in several languages, and on photographs of the bridge, although the dictionary was trained only on text."}]},{"t":"p","runs":[{"text":"The demonstration that made the idea public was a model with one feature amplified (the bridge) that then brought the bridge into every answer, including its account of its own identity. Behaviour followed the direction, which is the steering lesson to come."}]},{"t":"p","runs":[{"text":"Two free resources make all of this something you can do rather than read about. Gemma Scope publishes trained dictionaries for every layer of an open 2-billion-parameter model. Neuronpedia lets you browse the features in a browser, see what each fires on, and run your own text through them. Loading a dictionary and listing the features active on a sentence is twenty lines with the "},{"text":"sae_lens","code":true},{"text":" library in a free notebook."}]},{"t":"h2","id":"reading-a-feature","runs":[{"text":"Reading a feature"}]},{"t":"p","runs":[{"text":"A feature is described three ways. Its "},{"text":"top-activating examples","b":true},{"text":" (the passages where it fires hardest) give it a name. Its "},{"text":"logit effect","b":true},{"text":" (which tokens become more likely when its direction is added) says what it does to the output. And "},{"text":"steering","b":true},{"text":" (adding or removing it and watching the behaviour) says whether the name and the effect are real."}]},{"t":"p","runs":[{"text":"One property surprises everyone. Train a bigger dictionary on the same model and a single “birds” feature splits into species. Bigger again and the species split into contexts: a bird in a poem, a bird in a field guide. Features are not a fixed inventory the model has; they are a decomposition at a resolution you chose. The dictionary size is a dial, not a discovery."}]},{"t":"h2","id":"the-limits-which-are-the-important-part","runs":[{"text":"The limits, which are the important part"}],"role":"limit"},{"t":"p","runs":[{"text":"Reconstruction is imperfect.","b":true},{"text":" Replace the model’s activations with the dictionary’s reconstruction and run the model: the loss rises measurably. Some fraction of the computation is not captured by any feature (the dictionary’s dark matter) and nobody can yet say what it does."}]},{"t":"p","runs":[{"text":"Many features resist interpretation","b":true},{"text":", and some dictionary entries are dead, never firing, or dense, firing on everything. The count of clean features is impressive and it is a fraction of the dictionary."}]},{"t":"p","runs":[{"text":"Legibility is not mechanism.","b":true},{"text":" The dictionary is a basis chosen for our convenience. A feature that reads clearly to a person may not be a unit the model computes with; it may be a projection of several such units that happen to align with a human concept."}]},{"t":"p","runs":[{"text":"On downstream tasks the record is mixed.","b":true},{"text":" When dictionary features have been compared with plain linear probes on detection tasks (does this prompt contain a harmful request, is this output in the wrong language) probes have often matched or beaten them. Sparse autoencoders are a genuine advance in "},{"text":"discovering","i":true},{"text":" what a model represents. They are not yet a reliable instrument for "},{"text":"auditing","i":true},{"text":" it, and claims that a model has been “checked” with them should be read with that distinction in mind."}]},{"t":"h2","id":"what-you-can-do-with-one-this-week","runs":[{"text":"What you can do with one this week"}]},{"t":"p","runs":[{"text":"Run your own prompts through a published dictionary and list the features that fire; it is the fastest way to see how a model carves up your domain. Use a feature as a cheap classifier in the forward pass. Check whether a “harmful request” feature fires on the jailbreak examples of module six, and whether the refusal that follows tracks it. None of that needs more than a free notebook, and it turns the previous four lessons into something you have seen with your own eyes."}]}],"check":{"question":[{"text":"In training a sparse autoencoder on a model’s activations, what does the sparsity penalty do that plain reconstruction would not?"}],"options":[{"text":[{"text":"It shortens training, because the autoencoder can skip updating dictionary entries that are currently zero."}],"correct":false,"tempting":[{"text":"Sparsity does allow some computational shortcuts, but that is a side effect of implementation, not the reason the penalty is there."}]},{"text":[{"text":"It filters noise out of the activations, so the reconstruction is closer to the model’s true signal than the raw activations were."}],"correct":false,"tempting":[{"text":"Backwards: the penalty makes reconstruction worse, not better, and the gap it opens is the dictionary’s dark matter, not removed noise."}]},{"text":[{"text":"It forces the hidden layer to reproduce the model’s own neurons one-to-one, giving the dictionary a stable basis that can be compared across models."}],"correct":false,"tempting":[{"text":"The dictionary is wider than the neuron basis precisely so as not to reproduce it, and its features do not transfer across models without retraining."}]},{"text":[{"text":"It makes each input explainable by few dictionary entries, which pulls overlapping directions apart into separate features."}],"correct":true,"tempting":null}],"order":[0,2,3,1],"why":null},"prev":{"slug":"probing-what-is-represented","title":"Probing: asking the activations a question","minutes":9},"next":{"slug":"activation-patching-and-circuits","title":"Patching: finding which parts do the work","minutes":9}}}