Why the same question gives different answers

The model does not choose the word

At each step the model produces probabilities over its whole vocabulary. It does not pick one. A separate, small piece of code called the sampler does that, and how the sampler behaves is a setting.

Always taking the single most likely token is called greedy decoding. It sounds ideal and is not: greedy text is flat, repetitive, and prone to getting stuck in loops, because the most likely next word after a phrase is often a word that leads back into the same phrase.

So samplers introduce controlled randomness, and two dials control it.

Temperature flattens or sharpens the probabilities before the draw. At 0, effectively always the top choice. Around 0.7, the default in most chat products, likely words are strongly favoured but alternatives get a look in. Above about 1.2, the distribution is flat enough that improbable words start appearing, and the text drifts towards nonsense.

Top-p, or nucleus sampling, takes a different approach: keep only the smallest set of tokens whose probabilities add up to p, commonly 0.9, and draw from those. Where the model is confident, that set may be one token. Where it is unsure, it may be forty. It adapts, which is why it is now more common than a fixed top-k cut.

What this means for your work

The dial should match the task, and most people never touch it.

• Extraction, classification, data cleaning, code, anything checkable: low temperature. You want the most likely answer, repeatable.
• Brainstorming, names, alternative phrasings, first drafts: higher. Variety is the product.
• Anything you will run more than once and compare: fix it low, or you are measuring the sampler.

Where is the dial? In APIs and local tools such as Ollama, LM Studio and llama.cpp it is a direct parameter. In consumer chat apps it is usually hidden, and asking for “the most likely answer” does not change it: the wording of your request cannot alter a setting outside the model. What you can do in a chat app is generate several times and look at the spread, which tells you how stable the answer is.

The uncomfortable detail

Setting temperature to 0 does not guarantee identical output, and this catches out people trying to test carefully.

On the hardware these models run on, the arithmetic is done in parallel and in low precision, and the order in which many small numbers are added can vary between runs depending on batching: how many other people’s requests were bundled with yours. Adding the same numbers in a different order produces very slightly different results, and where two tokens were nearly tied, a tiny difference flips the choice. From there the paths diverge completely.

On top of that, providers change which model actually serves a request. The same product name may route to different sizes depending on load, region or your plan, usually without announcement.

The habit this should create

Never conclude anything from one run.

This single rule separates people who evaluate systems well from people who are repeatedly surprised in production. One good answer proves the good answer was possible. One bad answer proves the bad one was. Neither tells you the rate, and the rate is what you will live with.

When you want to know whether a prompt works, run it five or ten times on the same input and count. If it succeeds nine times in ten, you have a system that fails weekly at a hundred uses a day. If it succeeds five times in ten, the demo you saw was a coin toss.

This also reframes a common complaint. “It worked yesterday and not today” is usually not a change in the model. It is the same variation, observed twice.

And a note on repetition

If output loops (repeating a phrase, restating the same point in slightly different words) the cause is usually a low temperature with no repetition penalty, and the cure is to raise the temperature slightly or, in a chat app, to start again with a more specific request. A model repeating itself is not confused. It has found a self-reinforcing groove in its own text, which is what greedy decoding does.