Training against answers you can check

The change of signal

Preference training optimises a learned approximation of what people like. Its ceiling is the quality of that approximation, and its characteristic failure is reward hacking.

For some tasks you do not need an approximation. A maths answer is right or wrong. Code passes the tests or does not. A logical derivation checks out or does not. A structured output matches the schema or does not.

Where that is true, the reward is a program. It cannot be flattered, it does not drift, and it cannot be gamed by writing longer. This is usually called reinforcement learning from verifiable rewards, and it is the main technical development behind the reasoning models that appeared from late 2024 onward.

What the training does

The shape is straightforward. Give the model a problem with a checkable answer. Let it generate a full attempt, including whatever working it wants to produce. Check the final answer. Reward the whole attempt accordingly. Repeat across a large set of problems.

Nobody supervises the intermediate steps. The model is free to work in whatever way earns correct answers, and what emerges is the interesting part: attempts get longer. The model begins to check itself, try a second approach after a dead end, and revisit earlier steps. None of that was demonstrated in the training data; it was found because it raises the probability of a correct final answer.

DeepSeek’s R1 work in early 2025 made this unusually legible by publishing the recipe and showing response length rising over training as the model discovered that thinking longer paid.

Thinking tokens

The visible consequence is that these models generate a block of reasoning before their answer, often thousands of tokens, sometimes hidden from the user and always billed.

The mechanism is the one from module three’s first lesson: compute per token is fixed, so more tokens is more computation. A model that produces 4,000 tokens of working has done vastly more computation than one that answers in 20. This is why the technique is described as scaling test-time compute: you are buying capability at the moment of use rather than during training.

The costs are real and should be stated plainly. Responses take much longer. Token bills rise several-fold. And for simple questions the extra tokens buy nothing, which is why current products expose a reasoning-effort setting and why routing easy questions to a cheaper path is a live engineering problem rather than a solved one.

Where it works and where it stops

Works: competition mathematics, algorithmic programming, formal logic, structured extraction, anything with an automatic checker. The gains on these have been large and are not in dispute.

Does not transfer cleanly: open-ended writing, judgement calls, questions of taste, and any domain where correctness cannot be programmatically determined. There is evidence of partial transfer (a model trained to reason on maths is somewhat better at reasoning elsewhere) and there is disagreement about how much. Do not assume a reasoning model is better at your task; measure it, because on some tasks it is worse, more expensive and slower.

Needs problems with checkable answers at scale, which is a real bottleneck. This is one reason synthetic problem generation has become an active area.

The honest uncertainties

Three things are unsettled and get stated too confidently in both directions.

Whether this is reasoning. The models produce text that reads as deliberation and reach answers they would otherwise miss. Whether the visible text is the computation or a partial narration of it is discussed in module eight, where the evidence on faithfulness is mixed.

How far it scales. Longer thinking helps, then flattens, and on some problems degrades. Where the ceiling sits per domain is not established.

Whether it teaches new capability or elicits existing capability. Some evidence suggests reinforcement learning of this kind mostly sharpens what the base model could already do occasionally rather than adding new ability. This is an active argument with results on both sides.

What is not in doubt is the practical picture: verifiable rewards produced a large, reproducible improvement on checkable tasks, several open implementations exist, and the technique is now standard. Everything beyond that deserves the word “probably”.