What an A/B tells you that your eval set cannot
Your eval set answers "is this better on the cases I chose, by the criteria I wrote". An A/B answers "do people behave differently". Both matter and neither substitutes. A rewrite can improve every criterion on your set and reduce the number of people who finish the flow, because it now sounds like a lawyer.
An honest A/B is mostly about decisions you make before it starts.
Write it down first, in one paragraph
Experiment: reply-rewrite-v4
Primary metric: share of conversations resolved without
a human handover
Minimum effect we would act on: 3 percentage points
Guardrails: p95 latency, complaint rate, moderation reports
Allocation: 50/50, new sessions only
Runs: 2026-09-08 to 2026-09-22. No decision before the 22nd.Five lines. They are the difference between an experiment and a story you tell afterwards.
Peeking is the common way this goes wrong
You plan two weeks. On day four the metric is ahead with p = 0.03. You ship.
That p-value assumed you would look once, at the end. Looking every day and stopping the first time you cross the line means you get many chances to cross it by luck, and the real false-positive rate for a daily-peeked two-week test lands somewhere around 20 to 30% rather than 5%. Most "our A/B said it worked but it did nothing in production" stories are this.
Two honest fixes. Fix the horizon and genuinely do not decide early. Or use a method built for continuous monitoring — sequential tests, always-valid confidence intervals — which trade a little sensitivity for the right to look whenever you want. Pick one before you start.
Guardrails, because your primary metric is a proxy
Resolution rate goes up. Also: median latency went from 900ms to 2.4s, complaints mentioning "robotic" doubled, and unsubscribes ticked up. You would never have seen those, because you were watching one number.
Pick two or three guardrails that represent the ways a win could be hollow, and check them even when the headline is good. Especially when the headline is good.
Segments: declare them before, not after
The result is flat. Someone slices the data and finds it worked for Android users in Brazil. That finding is almost certainly noise — slice enough ways and something is always significant. If you had a reason in advance to expect a difference for a segment, declare it in advance and analyse it. Anything found afterwards is a hypothesis for the next experiment, never a conclusion from this one.
When you do not have the traffic
Many products cannot run a meaningful A/B. Four hundred sessions a week will not detect a three-point change this quarter or next. Say so out loud rather than running an underpowered test and treating its noise as a signal.
What works at small scale:
- Interleaving, where both variants serve the same user and you compare which one gets used. Far more sensitive per user, though it only applies to ranked or multi-option outputs.
- A long, boring baseline. Six weeks of before, six weeks of after, with your eyes open about everything else that changed (a holiday, a marketing push, a competitor's outage).
- Ten conversations with real users. Not statistics, and enormously more informative than a test with no power. Ten people telling you the new replies feel like a form letter is a finding.
The least honest option is the underpowered test, because it produces a number, and a number outranks a hunch in most meetings even when the hunch is better evidence.
Before you move on