Reasoning that is pattern matching, and how to tell

The perturbation experiments

Take a standard school maths word problem. “Ravi has 12 apples and gives 5 to Meena. How many are left?” A capable model answers it. Now change the names and the numbers, keeping the structure identical. Accuracy drops: a little for the strongest models, several points for most. Now add one sentence that is plausible and irrelevant: “Three of the apples were slightly smaller than the others.” Accuracy drops again, and for many models the drop is large: tens of points on problems whose arithmetic did not change at all.

That is the design of a 2024 study that built a perturbed version of a popular maths benchmark, and its result has been reproduced widely. Similar work uses trivially simple puzzles (“Alice has three brothers and two sisters; how many sisters does Alice’s brother have?”), on which strong models fail at surprising rates, and fail differently when a number changes.

If the model were executing the arithmetic, the names would not matter and the extra sentence would be ignored. They do matter. So something else is going on, and this course has given you the tools to say what.

The mechanism

Module five: training produces a function that maps token patterns to likely continuations, by reducing loss over an enormous corpus. That corpus contains the standard benchmarks themselves, many times, and countless near-copies (homework sites, tutorials, forum answers) all in a small number of templates.

So the model has seen the template “N items, gives M away, how many left” thousands of times, with the solution attached. What it learned is partly a procedure and partly a very good recogniser of the template. Change the surface and you move the input away from the region of the distribution where the recogniser is confident. The further away, the more the answer degrades. An irrelevant clause is especially damaging because the templates never had one, and the model has learned that every number in the problem is used.

This is not all-or-nothing. Models plainly do some genuine composition: they solve problems that are not in any template. The honest statement is that a benchmark score mixes two contributions, recognition and procedure, in proportions you cannot see from the score.

Contamination, and what a benchmark number means

The same mechanism makes published benchmark numbers upper bounds. Test items leak into training data through the web; a model that has seen a problem and its answer is recalling, not solving. Providers try to remove exact matches from training sets, and paraphrases get through.

The perturbation gap (score on the original minus score on a rewritten version) is the number that tells you how much of the headline was recall. It is also a better predictor of how the model will do on your inputs, because your inputs are not in the benchmark either.

Chain of thought: help and harm

Module three explained why asking for working helps: each token is a step of computation, so a problem with serial steps gets more compute when the steps are written out. The perturbation results add the other half. Chain of thought helps most on exactly the tasks where a procedure exists to be followed: arithmetic, symbolic manipulation, multi-step lookup.

It can also hurt. A 2024 study identified tasks on which asking for step-by-step reasoning lowered accuracy, and they were tasks on which humans also do worse when forced to verbalise: implicit pattern learning, some kinds of visual and statistical judgement. Writing out a rationale for a judgement that was not made by rationale produces a plausible rationale and a worse judgement. Whether the written steps are the computation at all is the subject of module eight.

The test you run on your own task

Take thirty items from your real workload. For each, make three variants:

1. Rename every entity and change every number.
2. Add one sentence that is plausible in context and irrelevant to the answer.
3. Reorder the information, keeping the content.

Score the original and the variants. A drop under five points says the model is mostly executing a procedure on your task. A drop over twenty says it is mostly recognising, and your production inputs, which will not match the templates, will see the lower number. The gap belongs beside the headline score in any report.

What follows

Reasoning models narrow the gap and do not close it. Models trained with verifiable rewards, from module five, drop less under perturbation because the training signal punished template-matching that got the wrong answer. They still drop.

Retrieval does not help reasoning. Putting more documents in the context adds facts; it does not make the procedure more robust.

Tools do. Module four’s tool-calling lesson exists for this. A model that writes 12 - 5 for a calculator, or a few lines of Python for a run, hands the procedure to something that does not care about the names. This is the reliable path for anything numerical.

Formatting the input to look like the training distribution is a mechanism, not a cheat. Standard layouts, one fact per line, numbers stated plainly. You are moving the input toward the region where the recogniser and the procedure agree.

The honest state of the argument

“Do these models reason?” is not going to be settled here, partly because nobody has an operational definition that both sides accept. What is settled is narrower and more useful: performance is sensitive to surface features in ways an algorithm’s would not be, the sensitivity is measurable on any task in an afternoon, it shrinks with scale and with reinforcement training, and it has not reached zero in any model anyone has tested.