Run it five times

Why the same question gives different answers

Send an identical prompt twice and you usually get two different replies. This is not a fault. At each step the model produces a probability distribution over possible next tokens, and the product samples from it rather than always taking the most likely one. The setting that controls how adventurous that sampling is is called temperature: near zero it almost always takes the top choice and answers are nearly repeatable; higher and it wanders.

Consumer chat products generally run at a moderate temperature and do not let you change it. APIs and local runners do, and it is one of the more useful dials you will ever be handed.

What this does to your testing

It means a single run is a sample, not a measurement. Three things follow, and ignoring them is how people reach confident conclusions from nothing.

One bad answer proves very little. If a tool fails a prompt once, run it twice more before you write it down. A tool that fails a prompt three times out of three has a problem. A tool that fails once out of three may just have been unlucky, and you would not want to be judged on your third-best attempt either.

One brilliant answer proves even less. This is the direction people forget. A launch demo, a screenshot in a thread, a friend’s amazing result: all of these are one draw from a distribution, and the ones that get shared are the good tail by construction. Nobody screenshots the mediocre attempt.

Differences smaller than the variation are not differences. If a tool’s own answers to one prompt range across two points on your scale, then a one-point gap between two tools on that prompt tells you nothing at all.

The practical rule

For the two or three prompts that matter most to your decision, run each three times per tool. For the rest, once is fine. This is about ten extra minutes and it stops most of the wrong conclusions.

When you do it, watch for one specific pattern: does the tool vary in style, or in substance? Three answers that say the same thing in different words is a healthy model. Three answers that give three different dates for the same event is a model that does not know the date, and the variation is the tell. You can use this deliberately as a confidence check: ask the same factual question in three separate conversations, and if the answers disagree, you have found something you must verify.

Turning it off where you can

If you have API access or a local model, set the temperature explicitly.

# near-deterministic: extraction, classification, formatting, code
client.chat.completions.create(model=M, messages=msgs, temperature=0)

# some variety: drafting, brainstorming, alternatives
client.chat.completions.create(model=M, messages=msgs, temperature=0.8)

For anything where you want the same input to give the same output (pulling fields out of invoices, sorting tickets, generating code) run it near zero. It reduces the surprises without costing quality.

One honest caveat: temperature zero is not a guarantee of identical output. Providers run these models across large fleets of hardware where the order of floating-point operations varies between machines, so tiny numerical differences can occasionally change a token, and a single changed token changes everything after it. Near-deterministic is the accurate description. If you build something that assumes byte-identical replies, it will break eventually.

What the variation looks like across products

Products differ in how much they wander, and it is not only the temperature setting. A product that runs a router may serve you a different model on the second attempt, which produces variation far larger than sampling alone. If a tool’s three answers differ in structure, length and depth rather than only in wording, suspect the router rather than the dice.

You can sometimes see this directly. Where the product names the model under each answer, check whether it is the same name all three times. Where it does not, note the pattern and treat that product’s single answers with more caution than a product that is consistently serving one model.

The one place variation is the point

When you want options rather than an answer (names, headlines, approaches, ways to phrase a difficult message) the variation is the feature. Ask for five and pick one. Raising the temperature produces more interesting alternatives, and asking the same question three times in three separate conversations produces more variety than asking for three options in one, because the model in a single reply tries to make its options differ from each other in shallow ways.