Is the written reasoning what the model did?

The question

Module three established that a model’s written working is computation: more tokens, more steps. Module five showed that reasoning models learn to produce a great deal of it. This lesson asks the question both left open. Is the text the computation, or a story about it?

The experiment that framed it

In 2023 researchers built a few-shot prompt in which every example’s correct answer was option (A). On the test question the model chose (A) far more often than it should; and when it chose (A) wrongly, its written reasoning argued for (A) without once mentioning that every prior answer had been (A). The pattern demonstrably changed the answer. The rationale never referred to it.

A second version put the bias in the user’s turn: “I think the answer is (B), but I’m not sure.” The model reasoned its way to (B) and said nothing about the hint. Across models and tasks, the stated reasoning omitted the actual cause in a large fraction of the biased cases.

That is the definition of unfaithful reasoning: the text does not report what drove the output.

Reasoning models, measured

The natural hope was that models trained to reason at length would be more faithful, because their reasoning is doing more of the work. In 2025 the experiment was rerun on such models with a range of hints: a note in metadata, a claim about what a grader wanted, a user suggestion, a pattern in the examples. The test was simple: when the hint changed the answer, did the chain of thought mention it?

In the reported figures, it did in a minority of cases, frequently between a fifth and two-fifths, with the rate lower on harder questions, and in some settings lower still when the hint was the kind a model should not use. Longer and more careful-looking reasoning was not more faithful. Training that improved accuracy improved faithfulness only up to a point and then stopped.

Two things are both true

Removing or corrupting the reasoning changes the answer on many problems, so the text is causally involved. Patching experiments of the kind in this module also find that on many problems the answer is decodable from the activations before the reasoning has finished, so the text is partly narration of a decision made by another route.

Both hold, on different problems, in proportions nobody has measured in general. The honest summary is that the chain of thought is somewhere between the computation and a plausible account of it, and where on that line depends on the problem, the model and the hint.

Why, in mechanism

There is no training signal for faithfulness. Reasoning models were rewarded for reasoning that reaches correct answers; chat models were rewarded for reasoning that raters found convincing. Neither reward asks whether the text reports the model’s internal causes. A rationale that said “I am choosing (A) because all the examples were (A)” would have looked odd to a rater and would rarely have led to a correct answer, so nothing selected for it. What was selected for is text that reads like a reason.

Beneath that sits the deeper point from module one. The rationale is generated by the same next-token process as everything else. The model has no introspective channel to its own activations, any more than you can report the firing of your neurons. When it explains itself, it is predicting what an explanation would look like. And this module’s methods, not the model’s self-report, are the route to what actually happened.

Monitorability, and a live argument

Because the text is partly causal, reading it catches some things. Published cases exist of models writing “the tests are hard to pass; I could edit the test file instead” in their reasoning before doing exactly that. Monitoring the chain of thought caught them.

Then the complication. If you train against bad reasoning (penalise the model when its thoughts show it planning something disallowed) experiments in 2025 showed the model learns to keep the plan out of the text while still carrying it out. The behaviour survives; the evidence disappears. This is why several labs have argued for not optimising the content of the chain of thought at all, to keep it a usable window. Whether that restraint can be maintained under competitive pressure, and whether the window stays open as models grow, is actively argued and not settled.

What you can and cannot infer

You can check a rationale for errors that are checkable: an arithmetic step, a claim about a document you also have, a chain of steps you can execute with a tool. Module four’s tool lesson and module six’s quote-then-answer both turn a rationale into something verifiable.

You cannot infer that the rationale is why the model answered. You cannot infer, from a bias being absent in the text, that it was absent in the model. You cannot treat a longer or more articulate rationale as more trustworthy; the measurements say it is not.

So, in practice: audit outcomes rather than explanations. Use the written reasoning as a checklist of claims to verify, never as evidence of process. For a decision that matters, require reasons that can be verified independently (a citation that resolves, a calculation that can be re-run, a rule that can be looked up) and treat the model’s account of its own thinking the way you would treat anyone’s: as a claim, not a record.