{"course":{"slug":"how-llms-work","title":"How a Language Model Actually Works","url":"https://addaly.com/learn/how-llms-work","level":"intermediate","track":"start"},"lesson":{"slug":"prompt-sensitivity","title":"Why a comma changes the answer","url":"https://addaly.com/learn/how-llms-work/prompt-sensitivity","position":55,"of":78,"minutes":8,"answer":"Format is part of the input distribution (through tokens, learned layouts, position and near-ties) so a score is a range across equivalent formats, and a design that permutes.","keyIdea":"Format is part of the input distribution (through tokens, learned layouts, position and near-ties) so a score is a range across equivalent formats, and a design that permutes, constrains and ensembles is more robust than a search for the one prompt that works."},"body":{"format":"markdown","text":"## The measurements\n\nRun the same classification task with the prompt formatted two ways: `Answer: ` with a trailing space and `Answer:` without. Or options labelled A to D versus 1 to 4. Or a blank line between examples versus none. These are the same task by any human reading.\n\nStudies from 2023 and 2024 that tried this systematically found accuracy swings across equivalent formats of up to several tens of percentage points on the same model and the same items. Not on average (the average swing is smaller), but the range is that wide, and which format wins is not consistent across models.\n\nMultiple-choice questions have their own version. Move the correct option from position A to position D and accuracy changes, often by several points. Some models lean toward A, others toward C, and the lean differs between versions of the same model. The order of few-shot examples matters too, and so does whether the last example happened to share the answer with the test item.\n\nNone of this is mysterious once you have the first five modules. It is four mechanisms acting at once.\n\n## Four mechanisms\n\n**Tokens.** Module one: `Answer:` followed by a space and `Answer:` followed by a newline are different token sequences, and the model has seen them in different contexts at different frequencies. The trailing-whitespace problem from the tokenizer lesson is the extreme case.\n\n**Learned formats.** Module five and the previous lesson: fine-tuning data comes in specific layouts. A format that matches what the model was tuned on sits in the dense part of the distribution; one that does not sits further out, and everything degrades with distance. The chat template of module three is the largest instance of this: get it wrong and the model is answering in a format it has never seen.\n\n**Position.** Module four's position bias applies inside a question as well as across a long document. Option D is at a different position from option A, attended to through a different set of positional relationships, and the training data had its own statistics about which position tends to be correct.\n\n**Near-ties.** Module three's lesson on temperature zero: when two continuations are close in logit, a small nudge from any of the above flips the argmax. Format sensitivity is worst on exactly the items the model finds hardest, because those are the items with the closest ties.\n\n## Why \"the best prompt\" is a fragile idea\n\nA prompt tuned by trial on fifty items has been fitted to one model's quirks on those items. Three things then happen. The tuned prompt's advantage shrinks on new items, because some of it was noise. It shrinks or reverses on the next model version, because the quirks moved. And the team, having seen a nine-point gain, defends the prompt against change for a year.\n\nPrompt tuning is real and worth doing; the prompting course covers how. The mechanics here add a warning label: a gain under about five points on a small set is within the range that format noise alone produces, and should not be believed until it holds across variants.\n\n## What to do, mechanically\n\n- **Report a range.** Run each evaluation under three or four equivalent formats and report the spread with the mean. A model whose scores range from 71 to 84 is a model whose score is \"somewhere in the seventies\", and saying \"84\" is choosing the format that flattered it.\n- **Permute options.** For multiple choice, run every ordering of the options and average, or score each option in a fixed slot separately. This removes the position component entirely, at the cost of more calls.\n- **Use the format the model was tuned on.** The chat template, exactly. Standard instruction phrasing rather than inventive phrasing. This is the single largest reduction available and it costs nothing.\n- **Take format freedom away from the output.** Module three's constrained decoding: if the answer must be one of four labels, mask everything else. The model cannot then lose points to \"Answer: B.\" versus \"B\".\n- **Ensemble over prompts.** Module three's sampling-many-answers lesson varied the random seed. Varying the prompt format and voting is the same idea applied to a different source of noise, and it is often more effective, because format noise is larger than sampling noise for many tasks.\n- **Normalise inputs.** One apostrophe, one dash, no stray whitespace, consistent casing in labels. Boring, and it removes a class of flips you would otherwise chase for a day.\n\n## What this is not\n\nIt is not a bug that will be fixed in the next release. Newer and larger models are less sensitive, instruction tuning reduces it, and no model tested has reached zero, because the four mechanisms above are the same ones that make the model work. A system that follows formatting is a system that can be moved by formatting. The engineering response is to measure the movement and design so that it does not matter, rather than to search for the one prompt on which it does not happen."},"check":{"question":"Two prompts differ only in whether the options are labelled A to D or 1 to 4. On the same 200 items, accuracy differs by nine points. What is the best reading of that gap?","options":[{"text":"The two labellings sit at different distances from the formats the model was tuned on, and format is part of the input.","correct":true},{"text":"Numbered labels are tokenised as digits, which the model treats arithmetically and cannot use as category names.","correct":false,"tempting":"Stretches a real tokenisation fact into an invented rule; models use numbered lists as labels constantly, and the direction of the gap varies between models."},{"text":"One of the runs hit an unlucky random seed, and repeating both at temperature zero would remove the difference.","correct":false,"tempting":"Confuses sampling noise with format sensitivity; the gap is between two different input sequences and persists at temperature zero."},{"text":"Nine points on 200 items is far outside noise, so this is a genuine defect in how the model handles letter labels that should be reported to the provider as a bug and worked around until it is fixed.","correct":false,"tempting":"Right that the gap is not noise, wrong about the category: this is a property of every model tested, not a defect in one, and it will not be patched away."}]},"prev":{"slug":"reasoning-by-pattern","title":"Reasoning that is pattern matching, and how to tell","url":"https://addaly.com/learn/how-llms-work/reasoning-by-pattern"},"next":{"slug":"drift-between-versions","title":"Drift: the same name, a different model","url":"https://addaly.com/learn/how-llms-work/drift-between-versions"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Why a comma changes the answer', in How a Language Model Actually Works. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/how-llms-work/prompt-sensitivity (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Why a comma changes the answer","courseTitle":"How a Language Model Actually Works","module":{"index":6,"slug":"why-it-goes-wrong","title":"Why it goes wrong, by mechanism"},"answer":null,"keyIdea":"Format is part of the input distribution (through tokens, learned layouts, position and near-ties) so a score is a range across equivalent formats, and a design that permutes, constrains and ensembles is more robust than a search for the one prompt that works.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"the-measurements","runs":[{"text":"The measurements"}]},{"t":"p","runs":[{"text":"Run the same classification task with the prompt formatted two ways: "},{"text":"Answer: ","code":true},{"text":" with a trailing space and "},{"text":"Answer:","code":true},{"text":" without. Or options labelled A to D versus 1 to 4. Or a blank line between examples versus none. These are the same task by any human reading."}]},{"t":"p","runs":[{"text":"Studies from 2023 and 2024 that tried this systematically found accuracy swings across equivalent formats of up to several tens of percentage points on the same model and the same items. Not on average (the average swing is smaller), but the range is that wide, and which format wins is not consistent across models."}]},{"t":"p","runs":[{"text":"Multiple-choice questions have their own version. Move the correct option from position A to position D and accuracy changes, often by several points. Some models lean toward A, others toward C, and the lean differs between versions of the same model. The order of few-shot examples matters too, and so does whether the last example happened to share the answer with the test item."}]},{"t":"p","runs":[{"text":"None of this is mysterious once you have the first five modules. It is four mechanisms acting at once."}]},{"t":"h2","id":"four-mechanisms","runs":[{"text":"Four mechanisms"}]},{"t":"p","runs":[{"text":"Tokens.","b":true},{"text":" Module one: "},{"text":"Answer:","code":true},{"text":" followed by a space and "},{"text":"Answer:","code":true},{"text":" followed by a newline are different token sequences, and the model has seen them in different contexts at different frequencies. The trailing-whitespace problem from the tokenizer lesson is the extreme case."}]},{"t":"p","runs":[{"text":"Learned formats.","b":true},{"text":" Module five and the previous lesson: fine-tuning data comes in specific layouts. A format that matches what the model was tuned on sits in the dense part of the distribution; one that does not sits further out, and everything degrades with distance. The chat template of module three is the largest instance of this: get it wrong and the model is answering in a format it has never seen."}]},{"t":"p","runs":[{"text":"Position.","b":true},{"text":" Module four’s position bias applies inside a question as well as across a long document. Option D is at a different position from option A, attended to through a different set of positional relationships, and the training data had its own statistics about which position tends to be correct."}]},{"t":"p","runs":[{"text":"Near-ties.","b":true},{"text":" Module three’s lesson on temperature zero: when two continuations are close in logit, a small nudge from any of the above flips the argmax. Format sensitivity is worst on exactly the items the model finds hardest, because those are the items with the closest ties."}]},{"t":"h2","id":"why-the-best-prompt-is-a-fragile-idea","runs":[{"text":"Why “the best prompt” is a fragile idea"}]},{"t":"p","runs":[{"text":"A prompt tuned by trial on fifty items has been fitted to one model’s quirks on those items. Three things then happen. The tuned prompt’s advantage shrinks on new items, because some of it was noise. It shrinks or reverses on the next model version, because the quirks moved. And the team, having seen a nine-point gain, defends the prompt against change for a year."}]},{"t":"p","runs":[{"text":"Prompt tuning is real and worth doing; the prompting course covers how. The mechanics here add a warning label: a gain under about five points on a small set is within the range that format noise alone produces, and should not be believed until it holds across variants."}]},{"t":"h2","id":"what-to-do-mechanically","runs":[{"text":"What to do, mechanically"}]},{"t":"ul","items":[[{"text":"Report a range.","b":true},{"text":" Run each evaluation under three or four equivalent formats and report the spread with the mean. A model whose scores range from 71 to 84 is a model whose score is “somewhere in the seventies”, and saying “84” is choosing the format that flattered it."}],[{"text":"Permute options.","b":true},{"text":" For multiple choice, run every ordering of the options and average, or score each option in a fixed slot separately. This removes the position component entirely, at the cost of more calls."}],[{"text":"Use the format the model was tuned on.","b":true},{"text":" The chat template, exactly. Standard instruction phrasing rather than inventive phrasing. This is the single largest reduction available and it costs nothing."}],[{"text":"Take format freedom away from the output.","b":true},{"text":" Module three’s constrained decoding: if the answer must be one of four labels, mask everything else. The model cannot then lose points to “Answer: B.” versus “B”."}],[{"text":"Ensemble over prompts.","b":true},{"text":" Module three’s sampling-many-answers lesson varied the random seed. Varying the prompt format and voting is the same idea applied to a different source of noise, and it is often more effective, because format noise is larger than sampling noise for many tasks."}],[{"text":"Normalise inputs.","b":true},{"text":" One apostrophe, one dash, no stray whitespace, consistent casing in labels. Boring, and it removes a class of flips you would otherwise chase for a day."}]]},{"t":"h2","id":"what-this-is-not","runs":[{"text":"What this is not"}]},{"t":"p","runs":[{"text":"It is not a bug that will be fixed in the next release. Newer and larger models are less sensitive, instruction tuning reduces it, and no model tested has reached zero, because the four mechanisms above are the same ones that make the model work. A system that follows formatting is a system that can be moved by formatting. The engineering response is to measure the movement and design so that it does not matter, rather than to search for the one prompt on which it does not happen."}]}],"check":{"question":[{"text":"Two prompts differ only in whether the options are labelled A to D or 1 to 4. On the same 200 items, accuracy differs by nine points. What is the best reading of that gap?"}],"options":[{"text":[{"text":"The two labellings sit at different distances from the formats the model was tuned on, and format is part of the input."}],"correct":true,"tempting":null},{"text":[{"text":"Numbered labels are tokenised as digits, which the model treats arithmetically and cannot use as category names."}],"correct":false,"tempting":[{"text":"Stretches a real tokenisation fact into an invented rule; models use numbered lists as labels constantly, and the direction of the gap varies between models."}]},{"text":[{"text":"One of the runs hit an unlucky random seed, and repeating both at temperature zero would remove the difference."}],"correct":false,"tempting":[{"text":"Confuses sampling noise with format sensitivity; the gap is between two different input sequences and persists at temperature zero."}]},{"text":[{"text":"Nine points on 200 items is far outside noise, so this is a genuine defect in how the model handles letter labels that should be reported to the provider as a bug and worked around until it is fixed."}],"correct":false,"tempting":[{"text":"Right that the gap is not noise, wrong about the category: this is a property of every model tested, not a defect in one, and it will not be patched away."}]}],"order":[1,3,0,2],"why":null},"prev":{"slug":"reasoning-by-pattern","title":"Reasoning that is pattern matching, and how to tell","minutes":9},"next":{"slug":"drift-between-versions","title":"Drift: the same name, a different model","minutes":8}}}