{"course":{"slug":"how-llms-work","title":"How a Language Model Actually Works","url":"https://addaly.com/learn/how-llms-work","level":"intermediate","track":"start"},"lesson":{"slug":"reinforcement-learning-on-checkable-answers","title":"Training against answers you can check","url":"https://addaly.com/learn/how-llms-work/reinforcement-learning-on-checkable-answers","position":47,"of":78,"minutes":9,"answer":"When correctness can be verified by a program, the reward is exact rather than a learned proxy, and models trained this way learn to spend more tokens working before answering.","keyIdea":"When correctness can be verified by a program, the reward is exact rather than a learned proxy, and models trained this way learn to spend more tokens working before answering."},"body":{"format":"markdown","text":"## The change of signal\n\nPreference training optimises a learned approximation of what people like. Its ceiling is the quality of that approximation, and its characteristic failure is reward hacking.\n\nFor some tasks you do not need an approximation. A maths answer is right or wrong. Code passes the tests or does not. A logical derivation checks out or does not. A structured output matches the schema or does not.\n\nWhere that is true, the reward is a program. It cannot be flattered, it does not drift, and it cannot be gamed by writing longer. This is usually called reinforcement learning from verifiable rewards, and it is the main technical development behind the reasoning models that appeared from late 2024 onward.\n\n## What the training does\n\nThe shape is straightforward. Give the model a problem with a checkable answer. Let it generate a full attempt, including whatever working it wants to produce. Check the final answer. Reward the whole attempt accordingly. Repeat across a large set of problems.\n\nNobody supervises the intermediate steps. The model is free to work in whatever way earns correct answers, and what emerges is the interesting part: attempts get longer. The model begins to check itself, try a second approach after a dead end, and revisit earlier steps. None of that was demonstrated in the training data; it was found because it raises the probability of a correct final answer.\n\nDeepSeek's R1 work in early 2025 made this unusually legible by publishing the recipe and showing response length rising over training as the model discovered that thinking longer paid.\n\n## Thinking tokens\n\nThe visible consequence is that these models generate a block of reasoning before their answer, often thousands of tokens, sometimes hidden from the user and always billed.\n\nThe mechanism is the one from module three's first lesson: compute per token is fixed, so more tokens is more computation. A model that produces 4,000 tokens of working has done vastly more computation than one that answers in 20. This is why the technique is described as scaling test-time compute: you are buying capability at the moment of use rather than during training.\n\nThe costs are real and should be stated plainly. Responses take much longer. Token bills rise several-fold. And for simple questions the extra tokens buy nothing, which is why current products expose a reasoning-effort setting and why routing easy questions to a cheaper path is a live engineering problem rather than a solved one.\n\n## Where it works and where it stops\n\n**Works:** competition mathematics, algorithmic programming, formal logic, structured extraction, anything with an automatic checker. The gains on these have been large and are not in dispute.\n\n**Does not transfer cleanly:** open-ended writing, judgement calls, questions of taste, and any domain where correctness cannot be programmatically determined. There is evidence of partial transfer (a model trained to reason on maths is somewhat better at reasoning elsewhere) and there is disagreement about how much. Do not assume a reasoning model is better at your task; measure it, because on some tasks it is worse, more expensive and slower.\n\n**Needs problems with checkable answers at scale**, which is a real bottleneck. This is one reason synthetic problem generation has become an active area.\n\n## The honest uncertainties\n\nThree things are unsettled and get stated too confidently in both directions.\n\n**Whether this is reasoning.** The models produce text that reads as deliberation and reach answers they would otherwise miss. Whether the visible text is the computation or a partial narration of it is discussed in module eight, where the evidence on faithfulness is mixed.\n\n**How far it scales.** Longer thinking helps, then flattens, and on some problems degrades. Where the ceiling sits per domain is not established.\n\n**Whether it teaches new capability or elicits existing capability.** Some evidence suggests reinforcement learning of this kind mostly sharpens what the base model could already do occasionally rather than adding new ability. This is an active argument with results on both sides.\n\nWhat is not in doubt is the practical picture: verifiable rewards produced a large, reproducible improvement on checkable tasks, several open implementations exist, and the technique is now standard. Everything beyond that deserves the word \"probably\"."},"check":{"question":"A team switches to a reasoning model for open-ended marketing copy and finds it slower, more expensive, and no better. Why is this the expected result?","options":[{"text":"Reasoning models are tuned for conciseness, which conflicts with creative writing.","correct":false,"tempting":"Reasoning models often are terse in their final answers, so a style mismatch is an observation-based and plausible-sounding account."},{"text":"The extra thinking tokens crowd out the context available for the brief.","correct":false,"tempting":"Context pressure is real and thinking tokens do consume budget, which makes this a mechanically-flavoured explanation."},{"text":"Creative tasks need a higher temperature than reasoning models permit.","correct":false,"tempting":"Sampling settings genuinely differ on reasoning endpoints, so a constraint on temperature sounds like practical experience."},{"text":"The training signal came from tasks with programmatically checkable answers, and marketing copy has no such checker, so the capability that was sharpened is not the one being used.","correct":true}]},"prev":{"slug":"preference-training","title":"Preference training, from reward models to DPO","url":"https://addaly.com/learn/how-llms-work/preference-training"},"next":{"slug":"emergence-and-the-argument","title":"Emergent abilities, and the argument about whether they exist","url":"https://addaly.com/learn/how-llms-work/emergence-and-the-argument"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Training against answers you can check', in How a Language Model Actually Works. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/how-llms-work/reinforcement-learning-on-checkable-answers (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Training against answers you can check","courseTitle":"How a Language Model Actually Works","module":{"index":5,"slug":"where-the-weights-come-from","title":"Where the weights come from"},"answer":null,"keyIdea":"When correctness can be verified by a program, the reward is exact rather than a learned proxy, and models trained this way learn to spend more tokens working before answering.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"the-change-of-signal","runs":[{"text":"The change of signal"}]},{"t":"p","runs":[{"text":"Preference training optimises a learned approximation of what people like. Its ceiling is the quality of that approximation, and its characteristic failure is reward hacking."}]},{"t":"p","runs":[{"text":"For some tasks you do not need an approximation. A maths answer is right or wrong. Code passes the tests or does not. A logical derivation checks out or does not. A structured output matches the schema or does not."}]},{"t":"p","runs":[{"text":"Where that is true, the reward is a program. It cannot be flattered, it does not drift, and it cannot be gamed by writing longer. This is usually called reinforcement learning from verifiable rewards, and it is the main technical development behind the reasoning models that appeared from late 2024 onward."}]},{"t":"h2","id":"what-the-training-does","runs":[{"text":"What the training does"}]},{"t":"p","runs":[{"text":"The shape is straightforward. Give the model a problem with a checkable answer. Let it generate a full attempt, including whatever working it wants to produce. Check the final answer. Reward the whole attempt accordingly. Repeat across a large set of problems."}]},{"t":"p","runs":[{"text":"Nobody supervises the intermediate steps. The model is free to work in whatever way earns correct answers, and what emerges is the interesting part: attempts get longer. The model begins to check itself, try a second approach after a dead end, and revisit earlier steps. None of that was demonstrated in the training data; it was found because it raises the probability of a correct final answer."}]},{"t":"p","runs":[{"text":"DeepSeek’s R1 work in early 2025 made this unusually legible by publishing the recipe and showing response length rising over training as the model discovered that thinking longer paid."}]},{"t":"h2","id":"thinking-tokens","runs":[{"text":"Thinking tokens"}]},{"t":"p","runs":[{"text":"The visible consequence is that these models generate a block of reasoning before their answer, often thousands of tokens, sometimes hidden from the user and always billed."}]},{"t":"p","runs":[{"text":"The mechanism is the one from module three’s first lesson: compute per token is fixed, so more tokens is more computation. A model that produces 4,000 tokens of working has done vastly more computation than one that answers in 20. This is why the technique is described as scaling test-time compute: you are buying capability at the moment of use rather than during training."}]},{"t":"p","runs":[{"text":"The costs are real and should be stated plainly. Responses take much longer. Token bills rise several-fold. And for simple questions the extra tokens buy nothing, which is why current products expose a reasoning-effort setting and why routing easy questions to a cheaper path is a live engineering problem rather than a solved one."}]},{"t":"h2","id":"where-it-works-and-where-it-stops","runs":[{"text":"Where it works and where it stops"}]},{"t":"p","runs":[{"text":"Works:","b":true},{"text":" competition mathematics, algorithmic programming, formal logic, structured extraction, anything with an automatic checker. The gains on these have been large and are not in dispute."}]},{"t":"p","runs":[{"text":"Does not transfer cleanly:","b":true},{"text":" open-ended writing, judgement calls, questions of taste, and any domain where correctness cannot be programmatically determined. There is evidence of partial transfer (a model trained to reason on maths is somewhat better at reasoning elsewhere) and there is disagreement about how much. Do not assume a reasoning model is better at your task; measure it, because on some tasks it is worse, more expensive and slower."}]},{"t":"p","runs":[{"text":"Needs problems with checkable answers at scale","b":true},{"text":", which is a real bottleneck. This is one reason synthetic problem generation has become an active area."}]},{"t":"h2","id":"the-honest-uncertainties","runs":[{"text":"The honest uncertainties"}]},{"t":"p","runs":[{"text":"Three things are unsettled and get stated too confidently in both directions."}]},{"t":"p","runs":[{"text":"Whether this is reasoning.","b":true},{"text":" The models produce text that reads as deliberation and reach answers they would otherwise miss. Whether the visible text is the computation or a partial narration of it is discussed in module eight, where the evidence on faithfulness is mixed."}]},{"t":"p","runs":[{"text":"How far it scales.","b":true},{"text":" Longer thinking helps, then flattens, and on some problems degrades. Where the ceiling sits per domain is not established."}]},{"t":"p","runs":[{"text":"Whether it teaches new capability or elicits existing capability.","b":true},{"text":" Some evidence suggests reinforcement learning of this kind mostly sharpens what the base model could already do occasionally rather than adding new ability. This is an active argument with results on both sides."}]},{"t":"p","runs":[{"text":"What is not in doubt is the practical picture: verifiable rewards produced a large, reproducible improvement on checkable tasks, several open implementations exist, and the technique is now standard. Everything beyond that deserves the word “probably”."}]}],"check":{"question":[{"text":"A team switches to a reasoning model for open-ended marketing copy and finds it slower, more expensive, and no better. Why is this the expected result?"}],"options":[{"text":[{"text":"Reasoning models are tuned for conciseness, which conflicts with creative writing."}],"correct":false,"tempting":[{"text":"Reasoning models often are terse in their final answers, so a style mismatch is an observation-based and plausible-sounding account."}]},{"text":[{"text":"The extra thinking tokens crowd out the context available for the brief."}],"correct":false,"tempting":[{"text":"Context pressure is real and thinking tokens do consume budget, which makes this a mechanically-flavoured explanation."}]},{"text":[{"text":"Creative tasks need a higher temperature than reasoning models permit."}],"correct":false,"tempting":[{"text":"Sampling settings genuinely differ on reasoning endpoints, so a constraint on temperature sounds like practical experience."}]},{"text":[{"text":"The training signal came from tasks with programmatically checkable answers, and marketing copy has no such checker, so the capability that was sharpened is not the one being used."}],"correct":true,"tempting":null}],"order":[2,1,0,3],"why":null},"prev":{"slug":"preference-training","title":"Preference training, from reward models to DPO","minutes":9},"next":{"slug":"emergence-and-the-argument","title":"Emergent abilities, and the argument about whether they exist","minutes":8}}}