What interpretability can and cannot yet tell you

The argument this module has been circling

In 2021 an influential paper described language models as stochastic parrots: systems that recombine the forms of text with no grip on what the text is about. The opposing claim, made with increasing confidence since, is that models build internal models of the world the text describes. Every lesson in this module is evidence for one side or the other, and it is time to say where the evidence leaves the argument.

For internal models: a network trained on Othello moves represents the board. Probes find linear coordinates for the latitude and longitude of cities and the dates of events in the activations of models that were only ever shown text. Dictionary features fire for the same concept across languages and across text and images. Circuits implement real algorithms. Attribution graphs show a model choosing a rhyme several tokens before it arrives, and planning toward it.

Against: those representations are partial and inconsistent. The same model that maps cities cannot reverse a fact it knows, drops twenty points when a word problem’s names change, and reports reasons for its answers that are not the causes. The map exists and the model does not always consult it.

The honest position is that models build partial, task-shaped internal models of what their text is about, that this is now demonstrated rather than conjectured, and that whether it deserves the word “understanding” is a dispute about the word. Both slogans (parrot, and mind) are stronger than the evidence.

What is settled

1. Features are directions, mostly linear, packed in superposition, with known non-linear exceptions.
2. Specific circuits implement specific behaviours, verified by causal intervention, completely in small models and partially in large ones.
3. Steering along a direction changes behaviour predictably, including refusal, which in open chat models is one direction.
4. Facts are retrieved as mid-layer lookups at the subject’s position and copied forward by attention; editing the lookup works narrowly and breaks in predictable ways.
5. Stated reasoning is partially faithful at best, with no training signal that would make it more so.
6. Memorisation is driven by duplication and is measurable; unlearning is mostly suppression.

Each of those is a sentence you could not have said before this module and can now defend.

What is not

1. Completeness. No method explains more than a fraction of any model’s computation. The dictionary’s dark matter, the circuit’s missing twenty per cent, the reasoning that was decided elsewhere: these are the current boundaries, and they are wide.
2. Scale. Full mechanistic accounts exist for models a thousandth the size of what you use. Production models have partial attribution graphs for chosen behaviours.
3. Auditing. Nobody can yet certify that a model lacks a property. Behavioural evaluation tests inputs somebody thought to try; interpretability could in principle test the weights, and cannot yet do so with the coverage a guarantee would need.
4. The tools’ own faithfulness. A clean feature may be an artefact of the dictionary. A bright cell in a patching grid depends on the baseline. The methods are calibrated against each other and none is ground truth.

What a practitioner can use this week

• The logit lens from module two, to watch an answer form layer by layer and see where a wrong one went wrong.
• A linear probe on a model you host, for free monitoring in the forward pass (distress, language, domain) with the shuffled-label control.
• Steering on an open model, for quick experiments in tone or language, and for screening fine-tuning data by how far it pushes a trait direction.
• Dictionary features as classifiers, from the published dictionaries, in a free notebook.
• Patching to locate a behaviour before attempting to change it.
• The extraction test on your own fine-tuning data before you ship a shared model.

And a reading habit. When a vendor says the model was “verified safe using interpretability”, ask which method, at what coverage, and whether the claim is “we found no X” (a behavioural or mechanistic search that came up empty) or “there is no X”, which no current method can establish. The first is worth something. The second is not yet available from anyone.

The course, in one paragraph

Text becomes tokens by a compression rule frozen on one corpus; tokens become vectors; attention moves information between positions and feed-forward layers look things up; a final vector becomes a score for every token; a token is sampled. The weights came from driving one loss down over a filtered web, then from a learned proxy for approval, then from rewards that could be checked. Every characteristic failure (invention, agreement, forgetting what it was told, brittleness, drift) is that machinery doing what it does. Every cost is bandwidth and memory. And inside, the machinery is directions and circuits, partly legible, more each year, never yet whole.

What you can do now that you could not before: predict what a model will do from its mechanism, diagnose why it did something else, budget what it will cost, and read a claim about it and know which kind of evidence would settle it.

Where to go

The evaluation course is how to measure what this course explained. The agents course is how to build with it. The mathematics course puts the linear algebra under the word “direction”. The interpretability tooling is free, the open models fit on the laptop you are reading this on, and every result in this module was reproduced by somebody with nothing more than that.