Same question, different answer

Ask twice, get two answers

Type a question into a chatbot. Open a new chat and type it again. Often the answers differ: a different opening, different examples, sometimes a different fact. Teachers meeting this for the first time assume the tool is broken or that they phrased something differently. Neither. This is how it works.

The model does not look up an answer. At each step it has a spread of possible next words, each with a probability, and it picks one: usually not always the single most likely, but a draw weighted by likelihood. That draw is random. Run it twice and you walk two slightly different paths through the same distribution, and they diverge more with every word.

Tools expose a control for this, called temperature. Low temperature means the model nearly always picks the most probable word, so answers become repetitive and stable. High temperature means more randomness, more variety, more nonsense. Consumer chatbots set a middle value and do not let you change it. Developer tools do.

Why it matters for a teacher

Consistency in marking. Give the model the same essay twice and it may give two different grades. A later lesson deals with marking properly; for now, know that a single run is one draw, not a verdict.

Reproducing a worksheet. The excellent set of questions you generated last week cannot be regenerated by typing the same prompt. Save the output, not just the prompt.

The apparent argument. Students who “prove” the AI is inconsistent by asking twice have discovered something true, and the first lesson’s prediction activity explains it: for the old woman walking slowly into the ______, the class produced four answers. Ask that class again tomorrow and the tally shifts. Same distribution, different draw.

Turning randomness into a check

The variation is also a tool. For a question you cannot easily verify, ask it three times in three fresh chats and compare.

• Three different answers. The question is on unreliable ground. The model does not have a stable pattern for it. Do not use any of the three; go to a source.
• Three matching answers. The pattern is stable. This is evidence that the model has seen the answer many times, the reliable-ground signal from earlier in this block.

But notice what agreement does and does not tell you. It tells you the model’s learned pattern is peaked. It does not tell you the peak is the truth. If most of the internet believes that we only use ten percent of our brains, or that the Great Wall is visible from space, the model reproduces that belief every single time, in every chat, with total consistency. Popular misconceptions are the most stable outputs it has.

So: disagreement is a red flag you can trust. Agreement is a green flag you cannot. That asymmetry is the whole lesson.

Asking differently, not just again

A stronger version of the check varies the question rather than repeating it. Ask for the date of an event; then, in a new chat, ask what happened in that year; then ask it to list events of that decade. If the same event appears in the same year across three different framings, the pattern is deep. If it moves, you have found a specific the model is reconstructing rather than recalling.

Another variation: ask a second tool. Different companies train on overlapping but different text, so a fact that two independent models agree on has a better claim than one model agreeing with itself. Better, not proven: they read much of the same internet.

For the classroom

This is one of the easiest properties of AI to demonstrate without a device. Print the same question asked three times, with the three answers. Ask the class to explain why they differ. A student who has done the tally activity will say: because each time it rolls the dice on the next word. That student understands more about these systems than most adults.

The limitation

Some products now cache or stabilise answers to common questions, so you may find a well-known query returns identically every time even though the mechanism underneath is still a draw. And a few offer a “deterministic” setting for developers. Neither changes the reasoning: identical output from one system is one source, however many times you read it.