Why the same prompt gives a different answer

The model does not choose a word

At each step the model produces a probability for every token in its vocabulary: tens of thousands of numbers, one per possible next chunk. Something outside the model then picks one. That picker is called the sampler, and its settings are why you can send an identical prompt twice and get two different answers.

The main setting is temperature. Low temperature sharpens the distribution towards whatever was already most likely; high temperature flattens it, giving unlikely tokens more of a chance. At temperature 0 the sampler takes the single most probable token every time, which is called greedy decoding. Typical chat products run somewhere around 0.7 to 1.0 by default, which is why they feel fluent and slightly unpredictable.

The second setting you will meet is top-p, or nucleus sampling. It restricts the choice to the smallest set of tokens whose probabilities add up to p, then samples inside that set. With top_p = 0.9, a token sitting in the bottom 10% of probability mass cannot be chosen at all, however high the temperature. Some interfaces also expose top_k, which keeps the k most likely tokens.

What this means for your work

Regenerating is a re-roll, not a repair. Pressing regenerate draws a fresh sample from the same distribution produced by the same prompt. If the prompt is the problem, every draw comes from the same wrong pile. That is the whole argument of a later lesson, and now you know its mechanism.

A prompt that worked once has not been tested. This is the practical lesson most people skip. If you are about to run a prompt over 200 customer messages, run it five times over the same three messages first. Variation you can see in five runs is variation you will get in two hundred. A prompt that gives the right format four times out of five will produce forty broken rows in your batch.

Match the temperature to the task, where you can. Most consumer chat windows do not let you change it, but developer playgrounds and APIs do:

client.messages.create(
    model="...",
    temperature=0,          # extraction, classification, formatting
    messages=[...],
)

Use a low temperature (0 to 0.3) when there is a right answer: pulling a date out of an invoice, labelling a message, converting a format. Use a higher one (0.8 to 1.2) when you want range: names, first drafts, ideas you will throw most of away. If you have ever complained that a model’s brainstorms all sound the same, the temperature is the first thing to reach for, not the prompt.

The caveat about determinism

Setting temperature to 0 makes the sampler deterministic. It does not reliably make the system deterministic.

Two runs at temperature 0 can still differ, and the reason is arithmetic rather than mystery. Providers run your request on GPUs alongside other people’s requests, batched together. The batch size changes the order in which floating-point numbers are added, floating-point addition is not exactly associative, and a difference far below the fifth decimal place can flip which of two near-tied tokens comes out on top. Once one token differs, everything after it can differ.

So temperature 0 buys you much less variation, not a guarantee. If you need a fixed answer (an audit trail, a test that must not flake) store the output, do not expect to regenerate it. Some providers offer a seed parameter and describe it as “best effort”, which is an unusually honest label.

A cheap experiment worth doing tonight

Take a prompt you use often. Run it five times without changing a character. Put the five outputs side by side and mark what stayed the same and what moved.

What stays is what your prompt actually determined. What moves is what you left to the sampler. If the thing that moves is the part you care about (the tone, the number of items, whether it included the caveat) you have found the next line to write, and no amount of regenerating would have found it for you.

Most people never do this, and it is the single fastest way to see the difference between a prompt that works and a prompt that got lucky.