Three points better, or three points of noise

The number that ships bad changes

You run the golden set on the current prompt: 78 percent. You run it on the new one: 81 percent. Three points better. You ship it.

A fortnight later someone re-runs the old prompt and gets 80. Nothing changed. The three points were never there.

This is the most common failure in applied prompt work, and it is not a statistics problem so much as a habit problem: the run produced a number, the number was bigger, and nobody asked what the measurement’s own variability was.

Pair everything

The first and largest improvement costs nothing. Run both variants on the same inputs, and compare item by item rather than comparing two aggregate percentages.

Why it matters so much: most of the variance in an evaluation comes from the items, not the variants. Some questions are hard and both variants fail them; some are easy and both pass. Those items contribute nothing to the comparison and, in an unpaired design, they contribute a great deal of noise.

Pairing cancels them. What is left is only the discordant items (where one variant passed and the other failed) and those are the entire evidence.

            new passes   new fails
old passes      142          9       <- b = 9
old fails        17         32       <- c = 17

The 142 and the 32 are irrelevant to the comparison. The question is whether 17 versus 9 is more than chance would produce. McNemar’s test answers exactly that, and for these numbers it is not convincing: with 26 discordant pairs split 17 to 9, the two-sided p-value is around 0.17. You are looking at a plausible three-point improvement with no evidence behind it.

A paired bootstrap gives the same answer with no formula to remember: resample the items with replacement a few thousand times, recompute the difference each time, and read off the interval. Twenty lines of NumPy, free, and it works for any metric rather than just binary pass or fail.

Roughly how many you need

The honest shape, without pretending to precision:

• 50 items: only very large effects, roughly ten points or more, are detectable. Most of what you test here will be inconclusive.
• 200 items: differences of about five points become visible, depending on how often the variants actually disagree.
• 1,000 items: two-point differences come within reach.

The exact numbers depend on the discordant rate, which you can only know after running. The practical consequence is worth stating plainly: if your set has 100 items, you cannot resolve a two-point difference, and no amount of re-running will change that. Either accept that small changes are unmeasurable and decide on other grounds (cost, latency, simplicity) or grow the set.

That is not a counsel of despair. It is permission to stop agonising over changes your instrument cannot see, and to spend the effort on changes large enough to matter.

Sampling noise is separate and also real

Even with fixed inputs, the model does not return identical output twice.

Set temperature to 0 for comparisons. This removes most of the variance and it does not remove all of it: floating-point non-associativity means results can differ with batch composition and hardware, so identical requests occasionally produce different tokens even at zero temperature. It is a small effect and it is not zero, and it is why a rerun of an unchanged configuration does not always reproduce exactly.

If your product runs at a higher temperature, evaluate there and run each item several times, scoring the mean. That multiplies your evaluation cost by the number of samples, which is the real reason people skip it.

Three practices that cost nothing

Run the baseline twice. Same configuration, two runs. The difference between them is your noise floor, measured rather than assumed. Any candidate improvement smaller than that gap is not an improvement. This single habit prevents most bad ships and takes one extra run.

Change one thing. A variant that alters the prompt and k and the reranker tells you the bundle differs, and nothing about which part. When the bundle regresses six months later you will have no way back.

Report the interval. “81 percent (95% CI 76–86)” against “78 percent (73–83)” makes the situation legible to everyone in the room, including the person who wants to ship. A bare percentage invites a decision the data does not support, and the interval is two lines of bootstrap away.

The uncomfortable implication

Most published prompt-engineering advice (including techniques with names, diagrams and enthusiastic write-ups) has been evaluated on small sets, at one temperature, on one model, without a paired test or an interval.

That does not make the advice wrong. It makes it unverified, which is a different thing and should produce a different response: try it on your own set, with a baseline run for the noise floor, and see. Roughly half of what you try will turn out to do nothing measurable on your task, and knowing which half is worth more than any of the techniques individually.