{"course":{"slug":"evaluating-ai","title":"Knowing If It Works","url":"https://addaly.com/learn/evaluating-ai","level":"intermediate","track":"build"},"lesson":{"slug":"ab-testing-honestly","title":"A/B testing a feature honestly","url":"https://addaly.com/learn/evaluating-ai/ab-testing-honestly","position":55,"of":76,"minutes":8,"answer":"Fix the metric, the effect you would act on, and the stop date before you start; otherwise the result is a story.","keyIdea":"Fix the metric, the effect you would act on, and the stop date before you start; otherwise the result is a story."},"body":{"format":"markdown","text":"## What an A/B tells you that your eval set cannot\n\nYour eval set answers \"is this better on the cases I chose, by the criteria I wrote\". An A/B answers \"do people behave differently\". Both matter and neither substitutes. A rewrite can improve every criterion on your set and reduce the number of people who finish the flow, because it now sounds like a lawyer.\n\nAn honest A/B is mostly about decisions you make before it starts.\n\n## Write it down first, in one paragraph\n\n```\nExperiment: reply-rewrite-v4\nPrimary metric: share of conversations resolved without\n                a human handover\nMinimum effect we would act on: 3 percentage points\nGuardrails: p95 latency, complaint rate, moderation reports\nAllocation: 50/50, new sessions only\nRuns: 2026-09-08 to 2026-09-22. No decision before the 22nd.\n```\n\nFive lines. They are the difference between an experiment and a story you tell afterwards.\n\n## Peeking is the common way this goes wrong\n\nYou plan two weeks. On day four the metric is ahead with p = 0.03. You ship.\n\nThat p-value assumed you would look once, at the end. Looking every day and stopping the first time you cross the line means you get many chances to cross it by luck, and the real false-positive rate for a daily-peeked two-week test lands somewhere around 20 to 30% rather than 5%. Most \"our A/B said it worked but it did nothing in production\" stories are this.\n\nTwo honest fixes. Fix the horizon and do not decide early. Or use a method built for continuous monitoring (sequential tests, always-valid confidence intervals) which trade a little sensitivity for the right to look whenever you want. Pick one before you start.\n\n## Guardrails, because your primary metric is a proxy\n\nResolution rate goes up. Also: median latency went from 900ms to 2.4s, complaints mentioning \"robotic\" doubled, and unsubscribes ticked up. You would never have seen those, because you were watching one number.\n\nPick two or three guardrails that represent the ways a win could be hollow, and check them even when the headline is good. Especially when the headline is good.\n\n## Segments: declare them before, not after\n\nThe result is flat. Someone slices the data and finds it worked for Android users in Brazil. That finding is almost certainly noise: slice enough ways and something is always significant. If you had a reason in advance to expect a difference for a segment, declare it in advance and analyse it. Anything found afterwards is a hypothesis for the next experiment, never a conclusion from this one.\n\n## When you do not have the traffic\n\nMany products cannot run a meaningful A/B. Four hundred sessions a week will not detect a three-point change this quarter or next. Say so out loud rather than running an underpowered test and treating its noise as a signal.\n\nWhat works at small scale:\n\n- **Interleaving**, where both variants serve the same user and you compare which one gets used. Far more sensitive per user, though it only applies to ranked or multi-option outputs.\n- **A long, boring baseline.** Six weeks of before, six weeks of after, with your eyes open about everything else that changed (a holiday, a marketing push, a competitor's outage).\n- **Ten conversations with real users.** Not statistics, and enormously more informative than a test with no power. Ten people telling you the new replies feel like a form letter is a finding.\n\nThe least honest option is the underpowered test, because it produces a number, and a number outranks a hunch in most meetings even when the hunch is better evidence."},"check":{"question":"A team plans a two-week A/B and checks the dashboard daily. On day four the new variant is ahead at p = 0.03, so they stop and ship. What is the core error?","options":[{"text":"Stopping the moment a threshold is crossed changes what the threshold means, so their real chance of a false positive is much higher than 3%.","correct":true},{"text":"Nothing is wrong: significance is significance, and stopping early spares users the worse variant.","correct":false,"tempting":"Treats a p-value as a property of the data alone, independent of the rule that decided when to look at it."},{"text":"Four days is too short because weekday and weekend users behave differently.","correct":false,"tempting":"Names a genuine problem with short experiments that happens not to be the error being made here."},{"text":"p = 0.03 is too weak a threshold for a shipping decision; they should have required p < 0.01.","correct":false,"tempting":"Assumes a stricter cutoff repairs the damage, when repeated looks inflate the error rate at any cutoff."}]},"prev":{"slug":"error-analysis","title":"Reading fifty failures","url":"https://addaly.com/learn/evaluating-ai/error-analysis"},"next":{"slug":"shadow-and-canary-releases","title":"Getting evidence from live traffic without exposing users","url":"https://addaly.com/learn/evaluating-ai/shadow-and-canary-releases"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'A/B testing a feature honestly', in Knowing If It Works. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/evaluating-ai/ab-testing-honestly (Accessed: 31 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"A/B testing a feature honestly","courseTitle":"Knowing If It Works","module":{"index":6,"slug":"evaluation-in-the-loop","title":"Evaluation in the working loop"},"answer":null,"keyIdea":"Fix the metric, the effect you would act on, and the stop date before you start; otherwise the result is a story.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"what-an-a-b-tells-you-that-your-eval-set-cannot","runs":[{"text":"What an A/B tells you that your eval set cannot"}]},{"t":"p","runs":[{"text":"Your eval set answers “is this better on the cases I chose, by the criteria I wrote”. An A/B answers “do people behave differently”. Both matter and neither substitutes. A rewrite can improve every criterion on your set and reduce the number of people who finish the flow, because it now sounds like a lawyer."}]},{"t":"p","runs":[{"text":"An honest A/B is mostly about decisions you make before it starts."}]},{"t":"h2","id":"write-it-down-first-in-one-paragraph","runs":[{"text":"Write it down first, in one paragraph"}]},{"t":"code","lang":"","label":"Text","text":"Experiment: reply-rewrite-v4\nPrimary metric: share of conversations resolved without\n                a human handover\nMinimum effect we would act on: 3 percentage points\nGuardrails: p95 latency, complaint rate, moderation reports\nAllocation: 50/50, new sessions only\nRuns: 2026-09-08 to 2026-09-22. No decision before the 22nd."},{"t":"p","runs":[{"text":"Five lines. They are the difference between an experiment and a story you tell afterwards."}]},{"t":"h2","id":"peeking-is-the-common-way-this-goes-wrong","runs":[{"text":"Peeking is the common way this goes wrong"}]},{"t":"p","runs":[{"text":"You plan two weeks. On day four the metric is ahead with p = 0.03. You ship."}]},{"t":"p","runs":[{"text":"That p-value assumed you would look once, at the end. Looking every day and stopping the first time you cross the line means you get many chances to cross it by luck, and the real false-positive rate for a daily-peeked two-week test lands somewhere around 20 to 30% rather than 5%. Most “our A/B said it worked but it did nothing in production” stories are this."}]},{"t":"p","runs":[{"text":"Two honest fixes. Fix the horizon and do not decide early. Or use a method built for continuous monitoring (sequential tests, always-valid confidence intervals) which trade a little sensitivity for the right to look whenever you want. Pick one before you start."}]},{"t":"h2","id":"guardrails-because-your-primary-metric-is-a-proxy","runs":[{"text":"Guardrails, because your primary metric is a proxy"}]},{"t":"p","runs":[{"text":"Resolution rate goes up. Also: median latency went from 900ms to 2.4s, complaints mentioning “robotic” doubled, and unsubscribes ticked up. You would never have seen those, because you were watching one number."}]},{"t":"p","runs":[{"text":"Pick two or three guardrails that represent the ways a win could be hollow, and check them even when the headline is good. Especially when the headline is good."}]},{"t":"h2","id":"segments-declare-them-before-not-after","runs":[{"text":"Segments: declare them before, not after"}]},{"t":"p","runs":[{"text":"The result is flat. Someone slices the data and finds it worked for Android users in Brazil. That finding is almost certainly noise: slice enough ways and something is always significant. If you had a reason in advance to expect a difference for a segment, declare it in advance and analyse it. Anything found afterwards is a hypothesis for the next experiment, never a conclusion from this one."}]},{"t":"h2","id":"when-you-do-not-have-the-traffic","runs":[{"text":"When you do not have the traffic"}]},{"t":"p","runs":[{"text":"Many products cannot run a meaningful A/B. Four hundred sessions a week will not detect a three-point change this quarter or next. Say so out loud rather than running an underpowered test and treating its noise as a signal."}]},{"t":"p","runs":[{"text":"What works at small scale:"}]},{"t":"ul","items":[[{"text":"Interleaving","b":true},{"text":", where both variants serve the same user and you compare which one gets used. Far more sensitive per user, though it only applies to ranked or multi-option outputs."}],[{"text":"A long, boring baseline.","b":true},{"text":" Six weeks of before, six weeks of after, with your eyes open about everything else that changed (a holiday, a marketing push, a competitor’s outage)."}],[{"text":"Ten conversations with real users.","b":true},{"text":" Not statistics, and enormously more informative than a test with no power. Ten people telling you the new replies feel like a form letter is a finding."}]]},{"t":"p","runs":[{"text":"The least honest option is the underpowered test, because it produces a number, and a number outranks a hunch in most meetings even when the hunch is better evidence."}]}],"check":{"question":[{"text":"A team plans a two-week A/B and checks the dashboard daily. On day four the new variant is ahead at p = 0.03, so they stop and ship. What is the core error?"}],"options":[{"text":[{"text":"Stopping the moment a threshold is crossed changes what the threshold means, so their real chance of a false positive is much higher than 3%."}],"correct":true,"tempting":null},{"text":[{"text":"Nothing is wrong: significance is significance, and stopping early spares users the worse variant."}],"correct":false,"tempting":[{"text":"Treats a p-value as a property of the data alone, independent of the rule that decided when to look at it."}]},{"text":[{"text":"Four days is too short because weekday and weekend users behave differently."}],"correct":false,"tempting":[{"text":"Names a genuine problem with short experiments that happens not to be the error being made here."}]},{"text":[{"text":"p = 0.03 is too weak a threshold for a shipping decision; they should have required p < 0.01."}],"correct":false,"tempting":[{"text":"Assumes a stricter cutoff repairs the damage, when repeated looks inflate the error rate at any cutoff."}]}],"order":[1,0,2,3],"why":null},"prev":{"slug":"error-analysis","title":"Reading fifty failures","minutes":9},"next":{"slug":"shadow-and-canary-releases","title":"Getting evidence from live traffic without exposing users","minutes":8}}}