{"course":{"slug":"how-llms-work","title":"How a Language Model Actually Works","url":"https://addaly.com/learn/how-llms-work","level":"intermediate","track":"start"},"lesson":{"slug":"what-interpretability-can-tell-you","title":"What interpretability can and cannot yet tell you","url":"https://addaly.com/learn/how-llms-work/what-interpretability-can-tell-you","position":78,"of":78,"minutes":9,"answer":"Models demonstrably build partial, task-shaped internal models of what their text describes, and no current method can show that a model lacks a property.","keyIdea":"Models demonstrably build partial, task-shaped internal models of what their text describes, and no current method can show that a model lacks a property, so a claim of verified absence is a claim nobody can yet make."},"body":{"format":"markdown","text":"## The argument this module has been circling\n\nIn 2021 an influential paper described language models as stochastic parrots: systems that recombine the forms of text with no grip on what the text is about. The opposing claim, made with increasing confidence since, is that models build internal models of the world the text describes. Every lesson in this module is evidence for one side or the other, and it is time to say where the evidence leaves the argument.\n\nFor internal models: a network trained on Othello moves represents the board. Probes find linear coordinates for the latitude and longitude of cities and the dates of events in the activations of models that were only ever shown text. Dictionary features fire for the same concept across languages and across text and images. Circuits implement real algorithms. Attribution graphs show a model choosing a rhyme several tokens before it arrives, and planning toward it.\n\nAgainst: those representations are partial and inconsistent. The same model that maps cities cannot reverse a fact it knows, drops twenty points when a word problem's names change, and reports reasons for its answers that are not the causes. The map exists and the model does not always consult it.\n\nThe honest position is that models build partial, task-shaped internal models of what their text is about, that this is now demonstrated rather than conjectured, and that whether it deserves the word \"understanding\" is a dispute about the word. Both slogans (parrot, and mind) are stronger than the evidence.\n\n## What is settled\n\n1. **Features are directions**, mostly linear, packed in superposition, with known non-linear exceptions.\n2. **Specific circuits implement specific behaviours**, verified by causal intervention, completely in small models and partially in large ones.\n3. **Steering along a direction changes behaviour predictably**, including refusal, which in open chat models is one direction.\n4. **Facts are retrieved as mid-layer lookups** at the subject's position and copied forward by attention; editing the lookup works narrowly and breaks in predictable ways.\n5. **Stated reasoning is partially faithful** at best, with no training signal that would make it more so.\n6. **Memorisation is driven by duplication and is measurable**; unlearning is mostly suppression.\n\nEach of those is a sentence you could not have said before this module and can now defend.\n\n> Figure: What the module settled, and what it did not. Settled: Features are directions, mostly linear, packed in superposition; Specific circuits implement specific behaviours, verified by intervention; Steering along a direction changes behaviour predictably, refusal included; Facts are mid-layer lookups at the subject's position, and edits break in known ways; Memorisation is driven by duplication and is measurable. Not settled: No method explains more than a fraction of any model's computation; Full accounts exist for models a thousandth the size of what you use; Nobody can certify that a model lacks a property; The methods are calibrated against each other and none is ground truth; Whether any of it deserves the word understanding. The distinction that matters when you read a vendor's claim: \"we found no X\" is a search that came up empty and is worth something; \"there is no X\" is a guarantee that no current method can produce.\n\n## What is not\n\n1. **Completeness.** No method explains more than a fraction of any model's computation. The dictionary's dark matter, the circuit's missing twenty per cent, the reasoning that was decided elsewhere: these are the current boundaries, and they are wide.\n2. **Scale.** Full mechanistic accounts exist for models a thousandth the size of what you use. Production models have partial attribution graphs for chosen behaviours.\n3. **Auditing.** Nobody can yet certify that a model *lacks* a property. Behavioural evaluation tests inputs somebody thought to try; interpretability could in principle test the weights, and cannot yet do so with the coverage a guarantee would need.\n4. **The tools' own faithfulness.** A clean feature may be an artefact of the dictionary. A bright cell in a patching grid depends on the baseline. The methods are calibrated against each other and none is ground truth.\n\n## What a practitioner can use this week\n\n- **The logit lens** from module two, to watch an answer form layer by layer and see where a wrong one went wrong.\n- **A linear probe** on a model you host, for free monitoring in the forward pass (distress, language, domain) with the shuffled-label control.\n- **Steering** on an open model, for quick experiments in tone or language, and for screening fine-tuning data by how far it pushes a trait direction.\n- **Dictionary features** as classifiers, from the published dictionaries, in a free notebook.\n- **Patching** to locate a behaviour before attempting to change it.\n- **The extraction test** on your own fine-tuning data before you ship a shared model.\n\nAnd a reading habit. When a vendor says the model was \"verified safe using interpretability\", ask which method, at what coverage, and whether the claim is \"we found no X\" (a behavioural or mechanistic search that came up empty) or \"there is no X\", which no current method can establish. The first is worth something. The second is not yet available from anyone.\n\n## The course, in one paragraph\n\nText becomes tokens by a compression rule frozen on one corpus; tokens become vectors; attention moves information between positions and feed-forward layers look things up; a final vector becomes a score for every token; a token is sampled. The weights came from driving one loss down over a filtered web, then from a learned proxy for approval, then from rewards that could be checked. Every characteristic failure (invention, agreement, forgetting what it was told, brittleness, drift) is that machinery doing what it does. Every cost is bandwidth and memory. And inside, the machinery is directions and circuits, partly legible, more each year, never yet whole.\n\nWhat you can do now that you could not before: predict what a model will do from its mechanism, diagnose why it did something else, budget what it will cost, and read a claim about it and know which kind of evidence would settle it.\n\n## Where to go\n\nThe evaluation course is how to measure what this course explained. The agents course is how to build with it. The mathematics course puts the linear algebra under the word \"direction\". The interpretability tooling is free, the open models fit on the laptop you are reading this on, and every result in this module was reproduced by somebody with nothing more than that."},"check":{"question":"A vendor states: \"Our interpretability team has verified that the model contains no capability to do X.\" Which reading is correct?","options":[{"text":"It is a strong mechanistic guarantee, because interpretability reads the weights directly rather than testing behaviour on sampled inputs.","correct":false,"tempting":"Reading weights directly is the aspiration; at current coverage every method explains a fraction of the computation, so absence in what was read is not absence in the model."},{"text":"It is meaningless, because interpretability is an academic exercise with no bearing on what a deployed model does.","correct":false,"tempting":"Dismisses real, causally verified results; the methods do bear on deployed behaviour, which is exactly why their limits are worth stating precisely."},{"text":"No current method has the coverage to show a capability is absent, so the claim can only mean that none was found.","correct":true},{"text":"It means a sparse autoencoder was trained on every layer and no dictionary entry corresponding to X was found, which at current dictionary sizes and reconstruction accuracy is sufficient to rule the capability out.","correct":false,"tempting":"Describes a real procedure and then overstates it; dictionaries miss a measurable share of the computation and their features are a chosen resolution, so an empty search is not a proof of absence."}]},"prev":{"slug":"unlearning","title":"Unlearning: why removing something is harder than adding it","url":"https://addaly.com/learn/how-llms-work/unlearning"},"next":null,"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'What interpretability can and cannot yet tell you', in How a Language Model Actually Works. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/how-llms-work/what-interpretability-can-tell-you (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"What interpretability can and cannot yet tell you","courseTitle":"How a Language Model Actually Works","module":{"index":8,"slug":"looking-inside","title":"Looking inside: what the weights actually hold"},"answer":null,"keyIdea":"Models demonstrably build partial, task-shaped internal models of what their text describes, and no current method can show that a model lacks a property, so a claim of verified absence is a claim nobody can yet make.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"left":{"title":"Settled","items":["Features are directions, mostly linear, packed in superposition","Specific circuits implement specific behaviours, verified by intervention","Steering along a direction changes behaviour predictably, refusal included","Facts are mid-layer lookups at the subject’s position, and edits break in known ways","Memorisation is driven by duplication and is measurable"]},"type":"compare","right":{"title":"Not settled","items":["No method explains more than a fraction of any model’s computation","Full accounts exist for models a thousandth the size of what you use","Nobody can certify that a model lacks a property","The methods are calibrated against each other and none is ground truth","Whether any of it deserves the word understanding"]},"title":"What the module settled, and what it did not","caption":"The distinction that matters when you read a vendor’s claim: “we found no X” is a search that came up empty and is worth something; “there is no X” is a guarantee that no current method can produce."},"alt":"Figure: What the module settled, and what it did not. Settled: Features are directions, mostly linear, packed in superposition; Specific circuits implement specific behaviours, verified by intervention; Steering along a direction changes behaviour predictably, refusal included; Facts are mid-layer lookups at the subject’s position, and edits break in known ways; Memorisation is driven by duplication and is measurable. Not settled: No method explains more than a fraction of any model’s computation; Full accounts exist for models a thousandth the size of what you use; Nobody can certify that a model lacks a property; The methods are calibrated against each other and none is ground truth; Whether any of it deserves the word understanding. The distinction that matters when you read a vendor’s claim: “we found no X” is a search that came up empty and is worth something; “there is no X” is a guarantee that no current method can produce."}},"blocks":[{"t":"h2","id":"the-argument-this-module-has-been-circling","runs":[{"text":"The argument this module has been circling"}]},{"t":"p","runs":[{"text":"In 2021 an influential paper described language models as stochastic parrots: systems that recombine the forms of text with no grip on what the text is about. The opposing claim, made with increasing confidence since, is that models build internal models of the world the text describes. Every lesson in this module is evidence for one side or the other, and it is time to say where the evidence leaves the argument."}]},{"t":"p","runs":[{"text":"For internal models: a network trained on Othello moves represents the board. Probes find linear coordinates for the latitude and longitude of cities and the dates of events in the activations of models that were only ever shown text. Dictionary features fire for the same concept across languages and across text and images. Circuits implement real algorithms. Attribution graphs show a model choosing a rhyme several tokens before it arrives, and planning toward it."}]},{"t":"p","runs":[{"text":"Against: those representations are partial and inconsistent. The same model that maps cities cannot reverse a fact it knows, drops twenty points when a word problem’s names change, and reports reasons for its answers that are not the causes. The map exists and the model does not always consult it."}]},{"t":"p","runs":[{"text":"The honest position is that models build partial, task-shaped internal models of what their text is about, that this is now demonstrated rather than conjectured, and that whether it deserves the word “understanding” is a dispute about the word. Both slogans (parrot, and mind) are stronger than the evidence."}]},{"t":"h2","id":"what-is-settled","runs":[{"text":"What is settled"}]},{"t":"ol","start":1,"items":[[{"text":"Features are directions","b":true},{"text":", mostly linear, packed in superposition, with known non-linear exceptions."}],[{"text":"Specific circuits implement specific behaviours","b":true},{"text":", verified by causal intervention, completely in small models and partially in large ones."}],[{"text":"Steering along a direction changes behaviour predictably","b":true},{"text":", including refusal, which in open chat models is one direction."}],[{"text":"Facts are retrieved as mid-layer lookups","b":true},{"text":" at the subject’s position and copied forward by attention; editing the lookup works narrowly and breaks in predictable ways."}],[{"text":"Stated reasoning is partially faithful","b":true},{"text":" at best, with no training signal that would make it more so."}],[{"text":"Memorisation is driven by duplication and is measurable","b":true},{"text":"; unlearning is mostly suppression."}]]},{"t":"p","runs":[{"text":"Each of those is a sentence you could not have said before this module and can now defend."}]},{"t":"h2","id":"what-is-not","runs":[{"text":"What is not"}]},{"t":"ol","start":1,"items":[[{"text":"Completeness.","b":true},{"text":" No method explains more than a fraction of any model’s computation. The dictionary’s dark matter, the circuit’s missing twenty per cent, the reasoning that was decided elsewhere: these are the current boundaries, and they are wide."}],[{"text":"Scale.","b":true},{"text":" Full mechanistic accounts exist for models a thousandth the size of what you use. Production models have partial attribution graphs for chosen behaviours."}],[{"text":"Auditing.","b":true},{"text":" Nobody can yet certify that a model "},{"text":"lacks","i":true},{"text":" a property. Behavioural evaluation tests inputs somebody thought to try; interpretability could in principle test the weights, and cannot yet do so with the coverage a guarantee would need."}],[{"text":"The tools’ own faithfulness.","b":true},{"text":" A clean feature may be an artefact of the dictionary. A bright cell in a patching grid depends on the baseline. The methods are calibrated against each other and none is ground truth."}]]},{"t":"h2","id":"what-a-practitioner-can-use-this-week","runs":[{"text":"What a practitioner can use this week"}]},{"t":"ul","items":[[{"text":"The logit lens","b":true},{"text":" from module two, to watch an answer form layer by layer and see where a wrong one went wrong."}],[{"text":"A linear probe","b":true},{"text":" on a model you host, for free monitoring in the forward pass (distress, language, domain) with the shuffled-label control."}],[{"text":"Steering","b":true},{"text":" on an open model, for quick experiments in tone or language, and for screening fine-tuning data by how far it pushes a trait direction."}],[{"text":"Dictionary features","b":true},{"text":" as classifiers, from the published dictionaries, in a free notebook."}],[{"text":"Patching","b":true},{"text":" to locate a behaviour before attempting to change it."}],[{"text":"The extraction test","b":true},{"text":" on your own fine-tuning data before you ship a shared model."}]]},{"t":"p","runs":[{"text":"And a reading habit. When a vendor says the model was “verified safe using interpretability”, ask which method, at what coverage, and whether the claim is “we found no X” (a behavioural or mechanistic search that came up empty) or “there is no X”, which no current method can establish. The first is worth something. The second is not yet available from anyone."}]},{"t":"h2","id":"the-course-in-one-paragraph","runs":[{"text":"The course, in one paragraph"}]},{"t":"p","runs":[{"text":"Text becomes tokens by a compression rule frozen on one corpus; tokens become vectors; attention moves information between positions and feed-forward layers look things up; a final vector becomes a score for every token; a token is sampled. The weights came from driving one loss down over a filtered web, then from a learned proxy for approval, then from rewards that could be checked. Every characteristic failure (invention, agreement, forgetting what it was told, brittleness, drift) is that machinery doing what it does. Every cost is bandwidth and memory. And inside, the machinery is directions and circuits, partly legible, more each year, never yet whole."}]},{"t":"p","runs":[{"text":"What you can do now that you could not before: predict what a model will do from its mechanism, diagnose why it did something else, budget what it will cost, and read a claim about it and know which kind of evidence would settle it."}]},{"t":"h2","id":"where-to-go","runs":[{"text":"Where to go"}]},{"t":"p","runs":[{"text":"The evaluation course is how to measure what this course explained. The agents course is how to build with it. The mathematics course puts the linear algebra under the word “direction”. The interpretability tooling is free, the open models fit on the laptop you are reading this on, and every result in this module was reproduced by somebody with nothing more than that."}]}],"check":{"question":[{"text":"A vendor states: “Our interpretability team has verified that the model contains no capability to do X.” Which reading is correct?"}],"options":[{"text":[{"text":"It is a strong mechanistic guarantee, because interpretability reads the weights directly rather than testing behaviour on sampled inputs."}],"correct":false,"tempting":[{"text":"Reading weights directly is the aspiration; at current coverage every method explains a fraction of the computation, so absence in what was read is not absence in the model."}]},{"text":[{"text":"It is meaningless, because interpretability is an academic exercise with no bearing on what a deployed model does."}],"correct":false,"tempting":[{"text":"Dismisses real, causally verified results; the methods do bear on deployed behaviour, which is exactly why their limits are worth stating precisely."}]},{"text":[{"text":"No current method has the coverage to show a capability is absent, so the claim can only mean that none was found."}],"correct":true,"tempting":null},{"text":[{"text":"It means a sparse autoencoder was trained on every layer and no dictionary entry corresponding to X was found, which at current dictionary sizes and reconstruction accuracy is sufficient to rule the capability out."}],"correct":false,"tempting":[{"text":"Describes a real procedure and then overstates it; dictionaries miss a measurable share of the computation and their features are a chosen resolution, so an empty search is not a proof of absence."}]}],"order":[0,1,3,2],"why":null},"prev":{"slug":"unlearning","title":"Unlearning: why removing something is harder than adding it","minutes":9},"next":null}}