Asking what pure chance would have produced

The question under every comparison

Prompt B scores 5 points above prompt A on the same 200 items. Before anything else, one question has to be answered: if the two prompts were identical in quality, how often would a gap that big appear anyway?

There are two ways to answer it. You can look up the right named test, check its assumptions, and hope you chose correctly. Or you can build the world where the null is true and count. The second is a permutation test, it takes ten lines, and it is very hard to get wrong in a way you would not notice.

The engine

You have per-item results for both systems. Under the null hypothesis, the label saying which result came from A and which from B carries no information, so swapping them within an item should change nothing systematic. Do exactly that, many times.

import numpy as np
rng = np.random.default_rng(0)

a = np.array(scores_a)          # one score per item, same order
b = np.array(scores_b)
observed = b.mean() - a.mean()  # e.g. +0.050

diffs = b - a
null = []
for _ in range(10_000):
    signs = rng.choice([-1, 1], size=len(diffs))
    null.append((diffs * signs).mean())

null = np.array(null)
p = (np.abs(null) >= abs(observed)).mean()
print(f"gap {observed:+.3f}, p = {p:.3f}")

Flipping the sign of each item’s difference is the same as swapping which system is called A. Run it and you get a distribution of gaps that chance alone produces: for 200 items with this much spread, typically a standard deviation around 0.021. Against that, an observed +0.050 sits well out in the tail, and roughly 1.5% of shuffles exceed it. That is your p-value, computed from your own data, with no assumption of normality and no table.

Why this beats remembering tests

The permutation approach works for any statistic you can compute. Median latency. Recall at 5. A weighted composite of three rubric checks. Cost per successful task. There is no closed-form test for most of these, and you do not need one: you only need to be able to compute the number and to know what the null would scramble.

It also forces you to state your design, which is the part that most often goes wrong. What you shuffle is your claim about what is exchangeable.

• Paired evaluation, same items through both systems: flip signs within each item, as above.
• Two separate groups of users: shuffle the group labels across users.
• Items grouped by customer: shuffle at the customer level, keeping each customer’s rows together.

Shuffling rows when the rows are grouped gives a confidently wrong answer, and the next lessons cover why. The virtue of writing the shuffle by hand is that the mistake becomes visible in code rather than hiding in a function’s assumptions.

The A/A test, which you should run first

Before trusting any comparison machinery, point it at two copies of the same system. Run configuration A twice, feed both result sets into the test, and see what it says.

The result should be an unremarkable gap and a p-value scattered anywhere between 0 and 1. If A against A produces a “significant” difference, you have found a bug, and it is nearly always one of five:

• The two runs used different subsets, because an item errored out in one of them and was silently dropped.
• Results were compared in the wrong order, so item 41’s A score sits against item 42’s B score.
• A cache served the first run and not the second.
• The set is ordered, and a truncated run covers a different mix.
• Non-determinism is larger than you assumed, which is not a bug but is a finding.

An A/A test costs one extra run and it has caught bad harnesses at essentially every team that has bothered.

What it cannot do

A simulation answers exactly the question you encoded, and nothing more.

• It cannot rescue a biased sample. If your 200 items over-represent one intent, permutation gives you a precise p-value about a set that does not resemble your traffic.
• It cannot tell you the effect is important. A gap can be reliably real and commercially worthless; report the size and its interval next to the p-value.
• It assumes the outcomes you fed it are comparable. Two runs at temperature 0.7, compared without accounting for run-to-run variance, will produce a small p for a difference that vanishes tomorrow.

Ten lines, numpy, and a laptop. For small sets you can do the same thing in a spreadsheet with a column of random signs, and it is worth doing once by hand to see the null distribution appear.