Next-token prediction, and where the reasoning comes from

The loop

Strip away the interface and this is the whole of generation:

tokens = encode(prompt)
while True:
    logits = model(tokens)[-1]     # one score per vocabulary entry
    next_id = sample(logits)       # pick one, see lesson 7
    tokens.append(next_id)
    if next_id == END_TOKEN:
        break

The model runs over the whole sequence, produces a score for each of its 100,000-odd vocabulary entries, one is chosen, it is appended, and the whole thing runs again. Every token you read cost one full pass through every block.

In practice the keys and values computed for earlier positions are cached, so pass number 200 does not redo the work of the first 199 positions. That is the KV cache, and it is why the first token of a reply is slow and the rest arrive steadily.

Why such a plain objective produces so much

"Predict the next token" sounds too weak to yield anything interesting. The counter-argument is about what prediction demands.

To predict the final word of a detective novel, you have to have tracked the suspects, the alibis, and the conventions of the genre. To predict the next line of a Python traceback, you need to know what the exception means. To predict the closing sentence of a Tamil news report about a court ruling, you need the language, the case, and the register. The task is trivially stated and arbitrarily deep. Anything that helps predict text is worth learning, and enough gets learned that the result behaves like understanding on a great many inputs.

That is a claim about capability, not about experience, and it is where honest people disagree about what to call it.

Fixed compute per token

Here is the constraint that explains most of what follows. A 40-block model does exactly 40 blocks of work per token. Not more for a hard token, not less for an easy one. The word "the" and the final digit of a difficult calculation get identical compute.

So a question that needs six steps of work cannot get six steps of work into one token. Something has to give.

Which is why "work step by step" helps

When the model writes out its steps, each written token is another full pass, and the result of that pass is written down in a place the next pass can read: the text itself. Twelve tokens of working is twelve times the compute, with the intermediate results externalised so they do not have to be held in a single vector.

That is the mechanism. Reasoning models make it explicit and train for it, generating a long internal working before the answer.

Now the honest part. The written steps are not guaranteed to be the cause of the answer. Experiments have shown models that were nudged toward a particular answer by something in the prompt, produced fluent reasoning that never mentioned the nudge, and arrived at the nudged answer. The visible chain can be a rationalisation rather than a transcript. Faithfulness of stated reasoning is an active research area and an unsolved one, so do not read a chain of thought as an audit trail.

Does it plan?

The strong claim — that the model never plans, it only writes one word at a time — has become harder to defend. Interpretability work on poetry has found evidence of a model settling on a rhyme word before writing the line that leads to it, with the intended word represented internally several tokens early.

So something like planning happens within a forward pass. What does not happen is planning carried between passes by anything other than the text and the cache. There is no scratch space that survives a token boundary and is hidden from you. If the model needs to hold something, it either holds it inside one pass or writes it down where you can see it.