What a chatbot is doing when it answers

One chunk at a time

Here is the actual mechanism, with nothing left out.

The model is given all the text so far: the instructions the company wrote, your message, and whatever it has already written in this reply. From that, it produces a score for every possible next chunk of text. Then one chunk is chosen, added to the end, and the whole thing runs again.

So far:  The capital of Kenya is

Next chunk, with the model's scores:
  " Nairobi"    94%
  " the"         2%
  " a"           1%
  ...tens of thousands of others sharing the rest

Pick Nairobi. Now the text reads The capital of Kenya is Nairobi, and the model runs again to choose what follows. A paragraph is a few hundred repetitions of that loop.

A chunk is usually a word or part of a word. The technical name is a token, and you can safely forget it.

Why it is not a lookup

There is no database being searched. Nairobi scored highly because in the vast quantity of text the model was trained on, that sequence of words is followed by Nairobi overwhelmingly often, and training pushed the dials until the model reflected that.

This is why it can answer questions that appear nowhere in any document, and why it can produce an answer with the exact shape of a fact and no fact inside it. Same mechanism, both times.

Some products do attach a real search engine, which changes things. When that happens you usually see links. If there are no links, nothing was looked up.

Why the same question gives different answers

The next chunk is not always the top-scoring one. It is sampled, with the higher scores more likely to be picked. That is deliberate: always taking the top choice produces flat, repetitive text.

So asking twice gives you two different answers, and neither is the real one. This is not a fault, and the variation is not a confidence reading either. It will re-word things it has cold just as readily as things it is fabricating.

The second stage, which explains the personality

Raw training on internet text gives you something that continues text. It does not give you something helpful, and it does not give you something that declines to explain how to make a nerve agent.

So there is a second stage. People are shown pairs of answers and asked which is better. Those judgements are used to tune the model toward the preferred kind of answer. This is where the politeness, the structure, the safety refusals and the willingness to answer questions rather than continue them all come from.

It has a side effect worth knowing. People rated agreeable answers highly, so models tend to fold when you push back. Tell one that it is wrong and it will often apologise and change its answer, including when its first answer was correct. That is the tuning showing, not a reconsideration.

What about the models that show their thinking

Newer systems produce a long stretch of working-out before the final answer. It genuinely helps, particularly on multi-step problems, because the intermediate text gives the next step something to build on.

But it is the same loop generating that text, one chunk at a time. It is not a window into a separate reasoning process happening somewhere else. Models have been caught producing reasonable-looking working that does not match the answer they then give. Useful, often, but not a receipt.