{"course":{"slug":"how-llms-work","title":"How a Language Model Actually Works","url":"https://addaly.com/learn/how-llms-work","level":"intermediate","track":"start"},"lesson":{"slug":"reasoning-by-pattern","title":"Reasoning that is pattern matching, and how to tell","url":"https://addaly.com/learn/how-llms-work/reasoning-by-pattern","position":54,"of":78,"minutes":9,"answer":"A benchmark score mixes recognising a familiar template with executing a procedure, and the drop in accuracy when names, numbers and an irrelevant sentence are changed is the measurement that separates the two.","keyIdea":"A benchmark score mixes recognising a familiar template with executing a procedure, and the drop in accuracy when names, numbers and an irrelevant sentence are changed is the measurement that separates the two."},"body":{"format":"markdown","text":"## The perturbation experiments\n\nTake a standard school maths word problem. \"Ravi has 12 apples and gives 5 to Meena. How many are left?\" A capable model answers it. Now change the names and the numbers, keeping the structure identical. Accuracy drops: a little for the strongest models, several points for most. Now add one sentence that is plausible and irrelevant: \"Three of the apples were slightly smaller than the others.\" Accuracy drops again, and for many models the drop is large: tens of points on problems whose arithmetic did not change at all.\n\nThat is the design of a 2024 study that built a perturbed version of a popular maths benchmark, and its result has been reproduced widely. Similar work uses trivially simple puzzles (\"Alice has three brothers and two sisters; how many sisters does Alice's brother have?\"), on which strong models fail at surprising rates, and fail differently when a number changes.\n\nIf the model were executing the arithmetic, the names would not matter and the extra sentence would be ignored. They do matter. So something else is going on, and this course has given you the tools to say what.\n\n## The mechanism\n\nModule five: training produces a function that maps token patterns to likely continuations, by reducing loss over an enormous corpus. That corpus contains the standard benchmarks themselves, many times, and countless near-copies (homework sites, tutorials, forum answers) all in a small number of templates.\n\nSo the model has seen the template \"N items, gives M away, how many left\" thousands of times, with the solution attached. What it learned is partly a procedure and partly a very good recogniser of the template. Change the surface and you move the input away from the region of the distribution where the recogniser is confident. The further away, the more the answer degrades. An irrelevant clause is especially damaging because the templates never had one, and the model has learned that every number in the problem is used.\n\nThis is not all-or-nothing. Models plainly do some genuine composition: they solve problems that are not in any template. The honest statement is that a benchmark score mixes two contributions, recognition and procedure, in proportions you cannot see from the score.\n\n## Contamination, and what a benchmark number means\n\nThe same mechanism makes published benchmark numbers upper bounds. Test items leak into training data through the web; a model that has seen a problem and its answer is recalling, not solving. Providers try to remove exact matches from training sets, and paraphrases get through.\n\nThe perturbation gap (score on the original minus score on a rewritten version) is the number that tells you how much of the headline was recall. It is also a better predictor of how the model will do on your inputs, because your inputs are not in the benchmark either.\n\n## Chain of thought: help and harm\n\nModule three explained why asking for working helps: each token is a step of computation, so a problem with serial steps gets more compute when the steps are written out. The perturbation results add the other half. Chain of thought helps most on exactly the tasks where a procedure exists to be followed: arithmetic, symbolic manipulation, multi-step lookup.\n\nIt can also hurt. A 2024 study identified tasks on which asking for step-by-step reasoning *lowered* accuracy, and they were tasks on which humans also do worse when forced to verbalise: implicit pattern learning, some kinds of visual and statistical judgement. Writing out a rationale for a judgement that was not made by rationale produces a plausible rationale and a worse judgement. Whether the written steps are the computation at all is the subject of module eight.\n\n## The test you run on your own task\n\nTake thirty items from your real workload. For each, make three variants:\n\n1. Rename every entity and change every number.\n2. Add one sentence that is plausible in context and irrelevant to the answer.\n3. Reorder the information, keeping the content.\n\nScore the original and the variants. A drop under five points says the model is mostly executing a procedure on your task. A drop over twenty says it is mostly recognising, and your production inputs, which will not match the templates, will see the lower number. The gap belongs beside the headline score in any report.\n\n## What follows\n\n**Reasoning models narrow the gap and do not close it.** Models trained with verifiable rewards, from module five, drop less under perturbation because the training signal punished template-matching that got the wrong answer. They still drop.\n\n**Retrieval does not help reasoning.** Putting more documents in the context adds facts; it does not make the procedure more robust.\n\n**Tools do.** Module four's tool-calling lesson exists for this. A model that writes `12 - 5` for a calculator, or a few lines of Python for a run, hands the procedure to something that does not care about the names. This is the reliable path for anything numerical.\n\n**Formatting the input to look like the training distribution is a mechanism, not a cheat.** Standard layouts, one fact per line, numbers stated plainly. You are moving the input toward the region where the recogniser and the procedure agree.\n\n## The honest state of the argument\n\n\"Do these models reason?\" is not going to be settled here, partly because nobody has an operational definition that both sides accept. What is settled is narrower and more useful: performance is sensitive to surface features in ways an algorithm's would not be, the sensitivity is measurable on any task in an afternoon, it shrinks with scale and with reinforcement training, and it has not reached zero in any model anyone has tested."},"check":{"question":"A model scores 94 per cent on a public maths benchmark and 71 per cent on the same problems after names and numbers are changed and one irrelevant sentence is added. What does the 23-point gap measure?","options":[{"text":"The model's arithmetic degrades on the larger numbers introduced by the rewrite, so the variant set is simply harder.","correct":false,"tempting":"Assumes the rewrite changed difficulty; the perturbation studies keep magnitudes comparable, and the irrelevant sentence adds no arithmetic at all."},{"text":"Ordinary run-to-run variation: a difference of that size is within the noise you would expect from sampling on a few hundred items.","correct":false,"tempting":"Sampling noise is real and worth checking, but 23 points on hundreds of items is far outside it, and the drop is systematic in one direction across models."},{"text":"The added sentence pushed the problems past the model's effective context length, so information at the start was lost.","correct":false,"tempting":"Position effects are real for long prompts; a word problem plus one sentence is a few dozen tokens, nowhere near any effective-context limit."},{"text":"How much of the original score came from matching familiar surface patterns rather than executing the procedure.","correct":true}]},"prev":{"slug":"the-reversal-curse","title":"What it means to know a fact: the reversal curse","url":"https://addaly.com/learn/how-llms-work/the-reversal-curse"},"next":{"slug":"prompt-sensitivity","title":"Why a comma changes the answer","url":"https://addaly.com/learn/how-llms-work/prompt-sensitivity"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Reasoning that is pattern matching, and how to tell', in How a Language Model Actually Works. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/how-llms-work/reasoning-by-pattern (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Reasoning that is pattern matching, and how to tell","courseTitle":"How a Language Model Actually Works","module":{"index":6,"slug":"why-it-goes-wrong","title":"Why it goes wrong, by mechanism"},"answer":null,"keyIdea":"A benchmark score mixes recognising a familiar template with executing a procedure, and the drop in accuracy when names, numbers and an irrelevant sentence are changed is the measurement that separates the two.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"the-perturbation-experiments","runs":[{"text":"The perturbation experiments"}]},{"t":"p","runs":[{"text":"Take a standard school maths word problem. “Ravi has 12 apples and gives 5 to Meena. How many are left?” A capable model answers it. Now change the names and the numbers, keeping the structure identical. Accuracy drops: a little for the strongest models, several points for most. Now add one sentence that is plausible and irrelevant: “Three of the apples were slightly smaller than the others.” Accuracy drops again, and for many models the drop is large: tens of points on problems whose arithmetic did not change at all."}]},{"t":"p","runs":[{"text":"That is the design of a 2024 study that built a perturbed version of a popular maths benchmark, and its result has been reproduced widely. Similar work uses trivially simple puzzles (“Alice has three brothers and two sisters; how many sisters does Alice’s brother have?”), on which strong models fail at surprising rates, and fail differently when a number changes."}]},{"t":"p","runs":[{"text":"If the model were executing the arithmetic, the names would not matter and the extra sentence would be ignored. They do matter. So something else is going on, and this course has given you the tools to say what."}]},{"t":"h2","id":"the-mechanism","runs":[{"text":"The mechanism"}]},{"t":"p","runs":[{"text":"Module five: training produces a function that maps token patterns to likely continuations, by reducing loss over an enormous corpus. That corpus contains the standard benchmarks themselves, many times, and countless near-copies (homework sites, tutorials, forum answers) all in a small number of templates."}]},{"t":"p","runs":[{"text":"So the model has seen the template “N items, gives M away, how many left” thousands of times, with the solution attached. What it learned is partly a procedure and partly a very good recogniser of the template. Change the surface and you move the input away from the region of the distribution where the recogniser is confident. The further away, the more the answer degrades. An irrelevant clause is especially damaging because the templates never had one, and the model has learned that every number in the problem is used."}]},{"t":"p","runs":[{"text":"This is not all-or-nothing. Models plainly do some genuine composition: they solve problems that are not in any template. The honest statement is that a benchmark score mixes two contributions, recognition and procedure, in proportions you cannot see from the score."}]},{"t":"h2","id":"contamination-and-what-a-benchmark-number-means","runs":[{"text":"Contamination, and what a benchmark number means"}]},{"t":"p","runs":[{"text":"The same mechanism makes published benchmark numbers upper bounds. Test items leak into training data through the web; a model that has seen a problem and its answer is recalling, not solving. Providers try to remove exact matches from training sets, and paraphrases get through."}]},{"t":"p","runs":[{"text":"The perturbation gap (score on the original minus score on a rewritten version) is the number that tells you how much of the headline was recall. It is also a better predictor of how the model will do on your inputs, because your inputs are not in the benchmark either."}]},{"t":"h2","id":"chain-of-thought-help-and-harm","runs":[{"text":"Chain of thought: help and harm"}]},{"t":"p","runs":[{"text":"Module three explained why asking for working helps: each token is a step of computation, so a problem with serial steps gets more compute when the steps are written out. The perturbation results add the other half. Chain of thought helps most on exactly the tasks where a procedure exists to be followed: arithmetic, symbolic manipulation, multi-step lookup."}]},{"t":"p","runs":[{"text":"It can also hurt. A 2024 study identified tasks on which asking for step-by-step reasoning "},{"text":"lowered","i":true},{"text":" accuracy, and they were tasks on which humans also do worse when forced to verbalise: implicit pattern learning, some kinds of visual and statistical judgement. Writing out a rationale for a judgement that was not made by rationale produces a plausible rationale and a worse judgement. Whether the written steps are the computation at all is the subject of module eight."}]},{"t":"h2","id":"the-test-you-run-on-your-own-task","runs":[{"text":"The test you run on your own task"}]},{"t":"p","runs":[{"text":"Take thirty items from your real workload. For each, make three variants:"}]},{"t":"ol","start":1,"items":[[{"text":"Rename every entity and change every number."}],[{"text":"Add one sentence that is plausible in context and irrelevant to the answer."}],[{"text":"Reorder the information, keeping the content."}]]},{"t":"p","runs":[{"text":"Score the original and the variants. A drop under five points says the model is mostly executing a procedure on your task. A drop over twenty says it is mostly recognising, and your production inputs, which will not match the templates, will see the lower number. The gap belongs beside the headline score in any report."}]},{"t":"h2","id":"what-follows","runs":[{"text":"What follows"}]},{"t":"p","runs":[{"text":"Reasoning models narrow the gap and do not close it.","b":true},{"text":" Models trained with verifiable rewards, from module five, drop less under perturbation because the training signal punished template-matching that got the wrong answer. They still drop."}]},{"t":"p","runs":[{"text":"Retrieval does not help reasoning.","b":true},{"text":" Putting more documents in the context adds facts; it does not make the procedure more robust."}]},{"t":"p","runs":[{"text":"Tools do.","b":true},{"text":" Module four’s tool-calling lesson exists for this. A model that writes "},{"text":"12 - 5","code":true},{"text":" for a calculator, or a few lines of Python for a run, hands the procedure to something that does not care about the names. This is the reliable path for anything numerical."}]},{"t":"p","runs":[{"text":"Formatting the input to look like the training distribution is a mechanism, not a cheat.","b":true},{"text":" Standard layouts, one fact per line, numbers stated plainly. You are moving the input toward the region where the recogniser and the procedure agree."}]},{"t":"h2","id":"the-honest-state-of-the-argument","runs":[{"text":"The honest state of the argument"}]},{"t":"p","runs":[{"text":"“Do these models reason?” is not going to be settled here, partly because nobody has an operational definition that both sides accept. What is settled is narrower and more useful: performance is sensitive to surface features in ways an algorithm’s would not be, the sensitivity is measurable on any task in an afternoon, it shrinks with scale and with reinforcement training, and it has not reached zero in any model anyone has tested."}]}],"check":{"question":[{"text":"A model scores 94 per cent on a public maths benchmark and 71 per cent on the same problems after names and numbers are changed and one irrelevant sentence is added. What does the 23-point gap measure?"}],"options":[{"text":[{"text":"The model’s arithmetic degrades on the larger numbers introduced by the rewrite, so the variant set is simply harder."}],"correct":false,"tempting":[{"text":"Assumes the rewrite changed difficulty; the perturbation studies keep magnitudes comparable, and the irrelevant sentence adds no arithmetic at all."}]},{"text":[{"text":"Ordinary run-to-run variation: a difference of that size is within the noise you would expect from sampling on a few hundred items."}],"correct":false,"tempting":[{"text":"Sampling noise is real and worth checking, but 23 points on hundreds of items is far outside it, and the drop is systematic in one direction across models."}]},{"text":[{"text":"The added sentence pushed the problems past the model’s effective context length, so information at the start was lost."}],"correct":false,"tempting":[{"text":"Position effects are real for long prompts; a word problem plus one sentence is a few dozen tokens, nowhere near any effective-context limit."}]},{"text":[{"text":"How much of the original score came from matching familiar surface patterns rather than executing the procedure."}],"correct":true,"tempting":null}],"order":[0,2,3,1],"why":null},"prev":{"slug":"the-reversal-curse","title":"What it means to know a fact: the reversal curse","minutes":8},"next":{"slug":"prompt-sensitivity","title":"Why a comma changes the answer","minutes":8}}}