Probing: asking the activations a question

The method

If features are directions, the simplest way to find one is to ask for it by name. Collect the model’s activations at some layer for a few thousand inputs. Label each input with the property you care about: positive or negative sentiment, past or present tense, true or false, which language. Train the smallest possible classifier, a logistic regression, to predict the label from the activation.

from sklearn.linear_model import LogisticRegression
probe = LogisticRegression(max_iter=2000).fit(acts_train, y_train)
print("layer", layer, "accuracy", probe.score(acts_test, y_test))

If the probe reads the property with high accuracy, the property is linearly represented at that layer. The probe’s weight vector is the direction. This is a linear probe, and keeping it linear is the point: a more powerful probe could compute the property itself from raw ingredients, and then you would have learned about the probe rather than the model.

Run it at every layer and you get a profile. Syntactic properties tend to be readable early and fade; semantic ones rise through the middle layers; properties tied to the specific next token peak near the end. The profile is a rough map of what the computation is doing where, obtained without opening a single weight.

The result that changed the argument

In 2022 a model was trained on nothing but sequences of moves from the board game Othello: no rules, no board, just move after move as tokens. The question was whether it had learned anything beyond move statistics.

A probe trained to read the board state from the activations succeeded. The model was tracking which squares were occupied and by whom, though it had never been shown a board. The first probe that worked was non-linear, which left room for doubt. A later analysis found a linear probe worked once the labels were recast from “black or white” to “mine or theirs”: the model’s own frame of reference, which is not the one a human would have guessed. That detail matters: the model may represent what you are looking for in coordinates you did not think to try.

Then the step that made it more than a curiosity. Researchers edited the represented board (flipped a square in the activations) and the model’s next move changed to one legal on the edited board. The representation was not merely present; it was being used.

The caveat that governs everything

That last step was necessary, because a probe alone cannot show use. Information about the input flows through the network whether or not the model does anything with it. A probe can read a property that the model carries passively, the way a courier carries a letter without reading it. High probe accuracy shows the property is decodable. It does not show the model consults it.

Three controls belong in any probing result:

• Shuffled labels. Train the same probe on randomly permuted labels. It should fail. If it does not, the probe is fitting noise, which happens when activations are wide and examples are few.
• A layer-zero baseline. Probe the raw token embeddings. If the property is already readable there, the model’s layers did not compute it; it was in the input.
• Intervention. Change the direction and see whether the output changes. The Othello move. Without it, the claim stops at “decodable”.

Truth probes

From 2023, several groups reported linear directions in middle layers that separate true statements from false ones, with accuracies of seventy to ninety per cent on the sets they were trained on. This connects to module six’s uncertainty lesson (an internal signal about correctness does exist) and it comes with the caveats you would now expect. The directions generalise poorly from one kind of statement to another. It is disputed whether they encode truth, or what the model would be willing to assert, or plausibility, which are different things that agree on easy examples. As a lie detector they are partial, and anyone selling one as complete is ahead of the evidence.

What a probe is good for now

Monitoring at no extra cost. A probe on a model you host reads its property during the forward pass that was happening anyway. “This user is in distress”, “this request is about code”, “this output is in the wrong language”: each is a linear readout that costs microseconds and no extra call.

Debugging representations. Does the model represent the negation in the customer’s sentence? At which layer does the language of the input become readable? Where does it lose track of who “she” refers to in a long passage? These are questions a probe profile answers directly.

Locating before editing. The lessons on steering and on where facts live both start from a direction, and a probe is the cheapest way to find one.

The free tooling is mature: TransformerLens and nnsight expose activations from open models with a few lines, scikit-learn trains the probe, and a 2-billion-parameter model on a free notebook is enough to reproduce most published probing results.

The limits, stated plainly

You need labels, so you can only find what you can name. A direction found in one model does not transfer to another; it is a fact about those weights. Results are per layer and per position, and the choice of position (the last token, the entity’s token, the mean) changes the answer. And the fundamental limit is the one above: a probe tells you a thing is readable, and the model may or may not be reading it. The next lesson finds directions without labels; the one after finds out which of them matter.