Why a comma changes the answer

The measurements

Run the same classification task with the prompt formatted two ways: Answer: with a trailing space and Answer: without. Or options labelled A to D versus 1 to 4. Or a blank line between examples versus none. These are the same task by any human reading.

Studies from 2023 and 2024 that tried this systematically found accuracy swings across equivalent formats of up to several tens of percentage points on the same model and the same items. Not on average (the average swing is smaller), but the range is that wide, and which format wins is not consistent across models.

Multiple-choice questions have their own version. Move the correct option from position A to position D and accuracy changes, often by several points. Some models lean toward A, others toward C, and the lean differs between versions of the same model. The order of few-shot examples matters too, and so does whether the last example happened to share the answer with the test item.

None of this is mysterious once you have the first five modules. It is four mechanisms acting at once.

Four mechanisms

Tokens. Module one: Answer: followed by a space and Answer: followed by a newline are different token sequences, and the model has seen them in different contexts at different frequencies. The trailing-whitespace problem from the tokenizer lesson is the extreme case.

Learned formats. Module five and the previous lesson: fine-tuning data comes in specific layouts. A format that matches what the model was tuned on sits in the dense part of the distribution; one that does not sits further out, and everything degrades with distance. The chat template of module three is the largest instance of this: get it wrong and the model is answering in a format it has never seen.

Position. Module four’s position bias applies inside a question as well as across a long document. Option D is at a different position from option A, attended to through a different set of positional relationships, and the training data had its own statistics about which position tends to be correct.

Near-ties. Module three’s lesson on temperature zero: when two continuations are close in logit, a small nudge from any of the above flips the argmax. Format sensitivity is worst on exactly the items the model finds hardest, because those are the items with the closest ties.

Why “the best prompt” is a fragile idea

A prompt tuned by trial on fifty items has been fitted to one model’s quirks on those items. Three things then happen. The tuned prompt’s advantage shrinks on new items, because some of it was noise. It shrinks or reverses on the next model version, because the quirks moved. And the team, having seen a nine-point gain, defends the prompt against change for a year.

Prompt tuning is real and worth doing; the prompting course covers how. The mechanics here add a warning label: a gain under about five points on a small set is within the range that format noise alone produces, and should not be believed until it holds across variants.

What to do, mechanically

• Report a range. Run each evaluation under three or four equivalent formats and report the spread with the mean. A model whose scores range from 71 to 84 is a model whose score is “somewhere in the seventies”, and saying “84” is choosing the format that flattered it.
• Permute options. For multiple choice, run every ordering of the options and average, or score each option in a fixed slot separately. This removes the position component entirely, at the cost of more calls.
• Use the format the model was tuned on. The chat template, exactly. Standard instruction phrasing rather than inventive phrasing. This is the single largest reduction available and it costs nothing.
• Take format freedom away from the output. Module three’s constrained decoding: if the answer must be one of four labels, mask everything else. The model cannot then lose points to “Answer: B.” versus “B”.
• Ensemble over prompts. Module three’s sampling-many-answers lesson varied the random seed. Varying the prompt format and voting is the same idea applied to a different source of noise, and it is often more effective, because format noise is larger than sampling noise for many tasks.
• Normalise inputs. One apostrophe, one dash, no stray whitespace, consistent casing in labels. Boring, and it removes a class of flips you would otherwise chase for a day.

What this is not

It is not a bug that will be fixed in the next release. Newer and larger models are less sensitive, instruction tuning reduces it, and no model tested has reached zero, because the four mechanisms above are the same ones that make the model work. A system that follows formatting is a system that can be moved by formatting. The engineering response is to measure the movement and design so that it does not matter, rather than to search for the one prompt on which it does not happen.