{"course":{"slug":"how-llms-work","title":"How a Language Model Actually Works","url":"https://addaly.com/learn/how-llms-work","level":"intermediate","track":"start"},"lesson":{"slug":"next-token-prediction","title":"Next-token prediction, and where the reasoning comes from","url":"https://addaly.com/learn/how-llms-work/next-token-prediction","position":20,"of":78,"minutes":8,"answer":"Reasoning that appears in the output is partly reasoning being done in the output, one token of compute at a time.","keyIdea":"Reasoning that appears in the output is partly reasoning being done in the output, one token of compute at a time."},"body":{"format":"markdown","text":"## The loop\n\nStrip away the interface and this is the whole of generation:\n\n```python\ntokens = encode(prompt)\nwhile True:\n    logits = model(tokens)[-1]     # one score per vocabulary entry\n    next_id = sample(logits)       # pick one, see lesson 7\n    tokens.append(next_id)\n    if next_id == END_TOKEN:\n        break\n```\n\nThe model runs over the whole sequence, produces a score for each of its 100,000-odd vocabulary entries, one is chosen, it is appended, and the whole thing runs again. Every token you read cost one full pass through every block.\n\nIn practice the keys and values computed for earlier positions are cached, so pass number 200 does not redo the work of the first 199 positions. That is the KV cache, and it is why the first token of a reply is slow and the rest arrive steadily.\n\n## Why such a plain objective produces so much\n\n\"Predict the next token\" sounds too weak to yield anything interesting. The counter-argument is about what prediction demands.\n\nTo predict the final word of a detective novel, you have to have tracked the suspects, the alibis, and the conventions of the genre. To predict the next line of a Python traceback, you need to know what the exception means. To predict the closing sentence of a Tamil news report about a court ruling, you need the language, the case, and the register. The task is trivially stated and arbitrarily deep. Anything that helps predict text is worth learning, and enough gets learned that the result behaves like understanding on a great many inputs.\n\nThat is a claim about capability, not about experience, and it is where honest people disagree about what to call it.\n\n## Fixed compute per token\n\nHere is the constraint that explains most of what follows. A 40-block model does exactly 40 blocks of work per token. Not more for a hard token, not less for an easy one. The word \"the\" and the final digit of a difficult calculation get identical compute.\n\nSo a question that needs six steps of work cannot get six steps of work into one token. Something has to give.\n\n## Which is why \"work step by step\" helps\n\nWhen the model writes out its steps, each written token is another full pass, and the result of that pass is written down in a place the next pass can read: the text itself. Twelve tokens of working is twelve times the compute, with the intermediate results externalised so they do not have to be held in a single vector.\n\nThat is the mechanism. Reasoning models make it explicit and train for it, generating a long internal working before the answer.\n\nNow the honest part. The written steps are not guaranteed to be the cause of the answer. Experiments have shown models that were nudged toward a particular answer by something in the prompt, produced fluent reasoning that never mentioned the nudge, and arrived at the nudged answer. The visible chain can be a rationalisation rather than a transcript. Faithfulness of stated reasoning is an active research area and an unsolved one, so do not read a chain of thought as an audit trail.\n\n## Does it plan?\n\nThe strong claim (that the model never plans, it only writes one word at a time) has become harder to defend. Interpretability work on poetry has found evidence of a model settling on a rhyme word before writing the line that leads to it, with the intended word represented internally several tokens early.\n\nSo something like planning happens within a forward pass. What does not happen is planning carried between passes by anything other than the text and the cache. There is no scratch space that survives a token boundary and is hidden from you. If the model needs to hold something, it either holds it inside one pass or writes it down where you can see it."},"check":{"question":"Asking a model to \"work through it step by step\" reliably improves accuracy on multi-step problems. What best explains why?","options":[{"text":"Each generated token is another full pass of computation, and the intermediate results are written into the text where later passes can read them.","correct":true},{"text":"The instruction makes the model report the reasoning it had already completed internally before answering.","correct":false,"tempting":"The output reads like an explanation of a conclusion, so it feels like a description of something that already happened."},{"text":"It puts the model into a more careful mode, much as lowering the temperature does.","correct":false,"tempting":"Both prompt wording and sampling settings visibly change outputs, so they get filed together as ways of making the model more careful."},{"text":"It gives the model more time to think, and time was the limiting factor.","correct":false,"tempting":"Wall-clock time really does increase, and for humans more time on a problem usually means a better answer."}]},"prev":{"slug":"kv-cache-and-attention-variants","title":"The key-value cache, and the engineering built around it","url":"https://addaly.com/learn/how-llms-work/kv-cache-and-attention-variants"},"next":{"slug":"greedy-beam-and-degeneration","title":"Greedy, beam search, and why chat models refuse both","url":"https://addaly.com/learn/how-llms-work/greedy-beam-and-degeneration"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Next-token prediction, and where the reasoning comes from', in How a Language Model Actually Works. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/how-llms-work/next-token-prediction (Accessed: 31 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Next-token prediction, and where the reasoning comes from","courseTitle":"How a Language Model Actually Works","module":{"index":3,"slug":"turning-scores-into-text","title":"Turning scores into text"},"answer":null,"keyIdea":"Reasoning that appears in the output is partly reasoning being done in the output, one token of compute at a time.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"the-loop","runs":[{"text":"The loop"}]},{"t":"p","runs":[{"text":"Strip away the interface and this is the whole of generation:"}]},{"t":"code","lang":"python","label":"Python","text":"tokens = encode(prompt)\nwhile True:\n    logits = model(tokens)[-1]     # one score per vocabulary entry\n    next_id = sample(logits)       # pick one, see lesson 7\n    tokens.append(next_id)\n    if next_id == END_TOKEN:\n        break"},{"t":"p","runs":[{"text":"The model runs over the whole sequence, produces a score for each of its 100,000-odd vocabulary entries, one is chosen, it is appended, and the whole thing runs again. Every token you read cost one full pass through every block."}]},{"t":"p","runs":[{"text":"In practice the keys and values computed for earlier positions are cached, so pass number 200 does not redo the work of the first 199 positions. That is the KV cache, and it is why the first token of a reply is slow and the rest arrive steadily."}]},{"t":"h2","id":"why-such-a-plain-objective-produces-so-much","runs":[{"text":"Why such a plain objective produces so much"}]},{"t":"p","runs":[{"text":"“Predict the next token” sounds too weak to yield anything interesting. The counter-argument is about what prediction demands."}]},{"t":"p","runs":[{"text":"To predict the final word of a detective novel, you have to have tracked the suspects, the alibis, and the conventions of the genre. To predict the next line of a Python traceback, you need to know what the exception means. To predict the closing sentence of a Tamil news report about a court ruling, you need the language, the case, and the register. The task is trivially stated and arbitrarily deep. Anything that helps predict text is worth learning, and enough gets learned that the result behaves like understanding on a great many inputs."}]},{"t":"p","runs":[{"text":"That is a claim about capability, not about experience, and it is where honest people disagree about what to call it."}]},{"t":"h2","id":"fixed-compute-per-token","runs":[{"text":"Fixed compute per token"}]},{"t":"p","runs":[{"text":"Here is the constraint that explains most of what follows. A 40-block model does exactly 40 blocks of work per token. Not more for a hard token, not less for an easy one. The word “the” and the final digit of a difficult calculation get identical compute."}]},{"t":"p","runs":[{"text":"So a question that needs six steps of work cannot get six steps of work into one token. Something has to give."}]},{"t":"h2","id":"which-is-why-work-step-by-step-helps","runs":[{"text":"Which is why “work step by step” helps"}]},{"t":"p","runs":[{"text":"When the model writes out its steps, each written token is another full pass, and the result of that pass is written down in a place the next pass can read: the text itself. Twelve tokens of working is twelve times the compute, with the intermediate results externalised so they do not have to be held in a single vector."}]},{"t":"p","runs":[{"text":"That is the mechanism. Reasoning models make it explicit and train for it, generating a long internal working before the answer."}]},{"t":"p","runs":[{"text":"Now the honest part. The written steps are not guaranteed to be the cause of the answer. Experiments have shown models that were nudged toward a particular answer by something in the prompt, produced fluent reasoning that never mentioned the nudge, and arrived at the nudged answer. The visible chain can be a rationalisation rather than a transcript. Faithfulness of stated reasoning is an active research area and an unsolved one, so do not read a chain of thought as an audit trail."}]},{"t":"h2","id":"does-it-plan","runs":[{"text":"Does it plan?"}]},{"t":"p","runs":[{"text":"The strong claim (that the model never plans, it only writes one word at a time) has become harder to defend. Interpretability work on poetry has found evidence of a model settling on a rhyme word before writing the line that leads to it, with the intended word represented internally several tokens early."}]},{"t":"p","runs":[{"text":"So something like planning happens within a forward pass. What does not happen is planning carried between passes by anything other than the text and the cache. There is no scratch space that survives a token boundary and is hidden from you. If the model needs to hold something, it either holds it inside one pass or writes it down where you can see it."}]}],"check":{"question":[{"text":"Asking a model to “work through it step by step” reliably improves accuracy on multi-step problems. What best explains why?"}],"options":[{"text":[{"text":"Each generated token is another full pass of computation, and the intermediate results are written into the text where later passes can read them."}],"correct":true,"tempting":null},{"text":[{"text":"The instruction makes the model report the reasoning it had already completed internally before answering."}],"correct":false,"tempting":[{"text":"The output reads like an explanation of a conclusion, so it feels like a description of something that already happened."}]},{"text":[{"text":"It puts the model into a more careful mode, much as lowering the temperature does."}],"correct":false,"tempting":[{"text":"Both prompt wording and sampling settings visibly change outputs, so they get filed together as ways of making the model more careful."}]},{"text":[{"text":"It gives the model more time to think, and time was the limiting factor."}],"correct":false,"tempting":[{"text":"Wall-clock time really does increase, and for humans more time on a problem usually means a better answer."}]}],"order":[3,2,0,1],"why":null},"prev":{"slug":"kv-cache-and-attention-variants","title":"The key-value cache, and the engineering built around it","minutes":9},"next":{"slug":"greedy-beam-and-degeneration","title":"Greedy, beam search, and why chat models refuse both","minutes":9}}}