{"course":{"slug":"how-llms-work","title":"How a Language Model Actually Works","url":"https://addaly.com/learn/how-llms-work","level":"intermediate","track":"start"},"lesson":{"slug":"probing-what-is-represented","title":"Probing: asking the activations a question","url":"https://addaly.com/learn/how-llms-work/probing-what-is-represented","position":70,"of":78,"minutes":9,"answer":"A linear probe shows that a property is decodable from a layer's activations.","keyIdea":"A linear probe shows that a property is decodable from a layer's activations, which is necessary but not sufficient for the model using it: only an intervention that changes the direction and moves the output shows use."},"body":{"format":"markdown","text":"## The method\n\nIf features are directions, the simplest way to find one is to ask for it by name. Collect the model's activations at some layer for a few thousand inputs. Label each input with the property you care about: positive or negative sentiment, past or present tense, true or false, which language. Train the smallest possible classifier, a logistic regression, to predict the label from the activation.\n\n```python\nfrom sklearn.linear_model import LogisticRegression\nprobe = LogisticRegression(max_iter=2000).fit(acts_train, y_train)\nprint(\"layer\", layer, \"accuracy\", probe.score(acts_test, y_test))\n```\n\nIf the probe reads the property with high accuracy, the property is linearly represented at that layer. The probe's weight vector *is* the direction. This is a **linear probe**, and keeping it linear is the point: a more powerful probe could compute the property itself from raw ingredients, and then you would have learned about the probe rather than the model.\n\nRun it at every layer and you get a profile. Syntactic properties tend to be readable early and fade; semantic ones rise through the middle layers; properties tied to the specific next token peak near the end. The profile is a rough map of what the computation is doing where, obtained without opening a single weight.\n\n## The result that changed the argument\n\nIn 2022 a model was trained on nothing but sequences of moves from the board game Othello: no rules, no board, just move after move as tokens. The question was whether it had learned anything beyond move statistics.\n\nA probe trained to read the board state from the activations succeeded. The model was tracking which squares were occupied and by whom, though it had never been shown a board. The first probe that worked was non-linear, which left room for doubt. A later analysis found a linear probe worked once the labels were recast from \"black or white\" to \"mine or theirs\": the model's own frame of reference, which is not the one a human would have guessed. That detail matters: the model may represent what you are looking for in coordinates you did not think to try.\n\nThen the step that made it more than a curiosity. Researchers **edited** the represented board (flipped a square in the activations) and the model's next move changed to one legal on the edited board. The representation was not merely present; it was being used.\n\n## The caveat that governs everything\n\nThat last step was necessary, because a probe alone cannot show use. Information about the input flows through the network whether or not the model does anything with it. A probe can read a property that the model carries passively, the way a courier carries a letter without reading it. High probe accuracy shows the property is **decodable**. It does not show the model consults it.\n\nThree controls belong in any probing result:\n\n- **Shuffled labels.** Train the same probe on randomly permuted labels. It should fail. If it does not, the probe is fitting noise, which happens when activations are wide and examples are few.\n- **A layer-zero baseline.** Probe the raw token embeddings. If the property is already readable there, the model's layers did not compute it; it was in the input.\n- **Intervention.** Change the direction and see whether the output changes. The Othello move. Without it, the claim stops at \"decodable\".\n\n## Truth probes\n\nFrom 2023, several groups reported linear directions in middle layers that separate true statements from false ones, with accuracies of seventy to ninety per cent on the sets they were trained on. This connects to module six's uncertainty lesson (an internal signal about correctness does exist) and it comes with the caveats you would now expect. The directions generalise poorly from one kind of statement to another. It is disputed whether they encode truth, or what the model would be willing to assert, or plausibility, which are different things that agree on easy examples. As a lie detector they are partial, and anyone selling one as complete is ahead of the evidence.\n\n## What a probe is good for now\n\n**Monitoring at no extra cost.** A probe on a model you host reads its property during the forward pass that was happening anyway. \"This user is in distress\", \"this request is about code\", \"this output is in the wrong language\": each is a linear readout that costs microseconds and no extra call.\n\n**Debugging representations.** Does the model represent the negation in the customer's sentence? At which layer does the language of the input become readable? Where does it lose track of who \"she\" refers to in a long passage? These are questions a probe profile answers directly.\n\n**Locating before editing.** The lessons on steering and on where facts live both start from a direction, and a probe is the cheapest way to find one.\n\nThe free tooling is mature: TransformerLens and nnsight expose activations from open models with a few lines, scikit-learn trains the probe, and a 2-billion-parameter model on a free notebook is enough to reproduce most published probing results.\n\n## The limits, stated plainly\n\nYou need labels, so you can only find what you can name. A direction found in one model does not transfer to another; it is a fact about those weights. Results are per layer and per position, and the choice of position (the last token, the entity's token, the mean) changes the answer. And the fundamental limit is the one above: a probe tells you a thing is readable, and the model may or may not be reading it. The next lesson finds directions without labels; the one after finds out which of them matter."},"check":{"question":"A linear probe reads a document's publication year from a model's layer-12 activations with 90 per cent accuracy. What has been established?","options":[{"text":"That the model uses the year when generating text, since a property this readable must be influencing the output.","correct":false,"tempting":"The intuitive leap, and the one the lesson warns against: readability shows the information is carried, and carrying is not consulting."},{"text":"That the model was trained on data labelled with publication years, because it could not otherwise have learned the property.","correct":false,"tempting":"Models learn properties like date from unlabelled text all the time (vocabulary, references and style all carry it) so no labels were needed for it to be decodable."},{"text":"That the year is linearly decodable at that layer, and nothing yet about whether it affects the output.","correct":true},{"text":"That the year is stored in a specific neuron at layer 12, which could be located by inspecting the probe's largest weight and ablated to remove the model's sense of date.","correct":false,"tempting":"Superposition rules this out: the probe's direction is spread across many neurons, its largest weight is not a special one, and ablating a single neuron would barely dent the feature."}]},"prev":{"slug":"superposition-and-features","title":"Superposition: more features than dimensions","url":"https://addaly.com/learn/how-llms-work/superposition-and-features"},"next":{"slug":"sparse-autoencoders","title":"Sparse autoencoders: a dictionary for the residual stream","url":"https://addaly.com/learn/how-llms-work/sparse-autoencoders"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Probing: asking the activations a question', in How a Language Model Actually Works. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/how-llms-work/probing-what-is-represented (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Probing: asking the activations a question","courseTitle":"How a Language Model Actually Works","module":{"index":8,"slug":"looking-inside","title":"Looking inside: what the weights actually hold"},"answer":null,"keyIdea":"A linear probe shows that a property is decodable from a layer’s activations, which is necessary but not sufficient for the model using it: only an intervention that changes the direction and moves the output shows use.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"the-method","runs":[{"text":"The method"}]},{"t":"p","runs":[{"text":"If features are directions, the simplest way to find one is to ask for it by name. Collect the model’s activations at some layer for a few thousand inputs. Label each input with the property you care about: positive or negative sentiment, past or present tense, true or false, which language. Train the smallest possible classifier, a logistic regression, to predict the label from the activation."}]},{"t":"code","lang":"python","label":"Python","text":"from sklearn.linear_model import LogisticRegression\nprobe = LogisticRegression(max_iter=2000).fit(acts_train, y_train)\nprint(\"layer\", layer, \"accuracy\", probe.score(acts_test, y_test))"},{"t":"p","runs":[{"text":"If the probe reads the property with high accuracy, the property is linearly represented at that layer. The probe’s weight vector "},{"text":"is","i":true},{"text":" the direction. This is a "},{"text":"linear probe","b":true},{"text":", and keeping it linear is the point: a more powerful probe could compute the property itself from raw ingredients, and then you would have learned about the probe rather than the model."}]},{"t":"p","runs":[{"text":"Run it at every layer and you get a profile. Syntactic properties tend to be readable early and fade; semantic ones rise through the middle layers; properties tied to the specific next token peak near the end. The profile is a rough map of what the computation is doing where, obtained without opening a single weight."}]},{"t":"h2","id":"the-result-that-changed-the-argument","runs":[{"text":"The result that changed the argument"}]},{"t":"p","runs":[{"text":"In 2022 a model was trained on nothing but sequences of moves from the board game Othello: no rules, no board, just move after move as tokens. The question was whether it had learned anything beyond move statistics."}]},{"t":"p","runs":[{"text":"A probe trained to read the board state from the activations succeeded. The model was tracking which squares were occupied and by whom, though it had never been shown a board. The first probe that worked was non-linear, which left room for doubt. A later analysis found a linear probe worked once the labels were recast from “black or white” to “mine or theirs”: the model’s own frame of reference, which is not the one a human would have guessed. That detail matters: the model may represent what you are looking for in coordinates you did not think to try."}]},{"t":"p","runs":[{"text":"Then the step that made it more than a curiosity. Researchers "},{"text":"edited","b":true},{"text":" the represented board (flipped a square in the activations) and the model’s next move changed to one legal on the edited board. The representation was not merely present; it was being used."}]},{"t":"h2","id":"the-caveat-that-governs-everything","runs":[{"text":"The caveat that governs everything"}]},{"t":"p","runs":[{"text":"That last step was necessary, because a probe alone cannot show use. Information about the input flows through the network whether or not the model does anything with it. A probe can read a property that the model carries passively, the way a courier carries a letter without reading it. High probe accuracy shows the property is "},{"text":"decodable","b":true},{"text":". It does not show the model consults it."}]},{"t":"p","runs":[{"text":"Three controls belong in any probing result:"}]},{"t":"ul","items":[[{"text":"Shuffled labels.","b":true},{"text":" Train the same probe on randomly permuted labels. It should fail. If it does not, the probe is fitting noise, which happens when activations are wide and examples are few."}],[{"text":"A layer-zero baseline.","b":true},{"text":" Probe the raw token embeddings. If the property is already readable there, the model’s layers did not compute it; it was in the input."}],[{"text":"Intervention.","b":true},{"text":" Change the direction and see whether the output changes. The Othello move. Without it, the claim stops at “decodable”."}]]},{"t":"h2","id":"truth-probes","runs":[{"text":"Truth probes"}]},{"t":"p","runs":[{"text":"From 2023, several groups reported linear directions in middle layers that separate true statements from false ones, with accuracies of seventy to ninety per cent on the sets they were trained on. This connects to module six’s uncertainty lesson (an internal signal about correctness does exist) and it comes with the caveats you would now expect. The directions generalise poorly from one kind of statement to another. It is disputed whether they encode truth, or what the model would be willing to assert, or plausibility, which are different things that agree on easy examples. As a lie detector they are partial, and anyone selling one as complete is ahead of the evidence."}]},{"t":"h2","id":"what-a-probe-is-good-for-now","runs":[{"text":"What a probe is good for now"}]},{"t":"p","runs":[{"text":"Monitoring at no extra cost.","b":true},{"text":" A probe on a model you host reads its property during the forward pass that was happening anyway. “This user is in distress”, “this request is about code”, “this output is in the wrong language”: each is a linear readout that costs microseconds and no extra call."}]},{"t":"p","runs":[{"text":"Debugging representations.","b":true},{"text":" Does the model represent the negation in the customer’s sentence? At which layer does the language of the input become readable? Where does it lose track of who “she” refers to in a long passage? These are questions a probe profile answers directly."}]},{"t":"p","runs":[{"text":"Locating before editing.","b":true},{"text":" The lessons on steering and on where facts live both start from a direction, and a probe is the cheapest way to find one."}]},{"t":"p","runs":[{"text":"The free tooling is mature: TransformerLens and nnsight expose activations from open models with a few lines, scikit-learn trains the probe, and a 2-billion-parameter model on a free notebook is enough to reproduce most published probing results."}]},{"t":"h2","id":"the-limits-stated-plainly","runs":[{"text":"The limits, stated plainly"}],"role":"limit"},{"t":"p","runs":[{"text":"You need labels, so you can only find what you can name. A direction found in one model does not transfer to another; it is a fact about those weights. Results are per layer and per position, and the choice of position (the last token, the entity’s token, the mean) changes the answer. And the fundamental limit is the one above: a probe tells you a thing is readable, and the model may or may not be reading it. The next lesson finds directions without labels; the one after finds out which of them matter."}]}],"check":{"question":[{"text":"A linear probe reads a document’s publication year from a model’s layer-12 activations with 90 per cent accuracy. What has been established?"}],"options":[{"text":[{"text":"That the model uses the year when generating text, since a property this readable must be influencing the output."}],"correct":false,"tempting":[{"text":"The intuitive leap, and the one the lesson warns against: readability shows the information is carried, and carrying is not consulting."}]},{"text":[{"text":"That the model was trained on data labelled with publication years, because it could not otherwise have learned the property."}],"correct":false,"tempting":[{"text":"Models learn properties like date from unlabelled text all the time (vocabulary, references and style all carry it) so no labels were needed for it to be decodable."}]},{"text":[{"text":"That the year is linearly decodable at that layer, and nothing yet about whether it affects the output."}],"correct":true,"tempting":null},{"text":[{"text":"That the year is stored in a specific neuron at layer 12, which could be located by inspecting the probe’s largest weight and ablated to remove the model’s sense of date."}],"correct":false,"tempting":[{"text":"Superposition rules this out: the probe’s direction is spread across many neurons, its largest weight is not a special one, and ablating a single neuron would barely dent the feature."}]}],"order":[0,1,2,3],"why":null},"prev":{"slug":"superposition-and-features","title":"Superposition: more features than dimensions","minutes":9},"next":{"slug":"sparse-autoencoders","title":"Sparse autoencoders: a dictionary for the residual stream","minutes":10}}}