{"course":{"slug":"evaluating-ai","title":"Knowing If It Works","url":"https://addaly.com/learn/evaluating-ai","level":"intermediate","track":"build"},"lesson":{"slug":"peeking-and-stopping-rules","title":"Watching the dashboard until it says yes","url":"https://addaly.com/learn/evaluating-ai/peeking-and-stopping-rules","position":27,"of":76,"minutes":9,"answer":"Checking an experiment repeatedly and stopping when it crosses the line turns a five per cent error rate into something far larger.","keyIdea":"Checking an experiment repeatedly and stopping when it crosses the line turns a five per cent error rate into something far larger, and the effect you stop on is overestimated because you stopped at a high point."},"body":{"format":"markdown","text":"## What repeated looking does\n\nAn online experiment runs for a fortnight. The team checks the dashboard every morning and will stop when the p-value drops below 0.05. This feels efficient and it silently destroys the error rate.\n\nThe mechanism is simple. A p-value computed on accumulating data wanders. Even when there is no effect at all, it drifts up and down, and over enough looks it will dip below 0.05 at some point. Checking five times pushes the false-positive rate from 5% to roughly 14%. Checking continuously, with no fixed horizon, means the probability of eventually crossing approaches certainty.\n\nYou can watch it happen in a few lines:\n\n```python\nimport numpy as np\nrng = np.random.default_rng(0)\nhits = 0\nfor _ in range(2000):\n    a = rng.random(4000) < 0.20        # identical systems\n    b = rng.random(4000) < 0.20\n    for n in range(400, 4001, 400):    # ten peeks\n        pa, pb = a[:n].mean(), b[:n].mean()\n        se = np.sqrt(2 * 0.2 * 0.8 / n)\n        if abs(pb - pa) > 1.96 * se:\n            hits += 1\n            break\nprint(hits / 2000)                      # ≈ 0.18\n```\n\nTwo identical systems, and eighteen per cent of experiments declare a winner.\n\n> Figure: The false-positive rate rises with every look. Line chart of Chance of a false winner, % against Number of peeks at the dashboard: Two identical systems, stop at the first p below 0.05 (1, 5) (2, 8) (3, 11) (5, 14) (10, 18). The nominal 5% holds for exactly one look. Ten looks at an experiment with no effect at all declare a winner about one time in five, and the effect reported at the crossing is biased upwards.\n\n## The second, quieter damage\n\nStopping at the moment of crossing does not only produce false positives. It biases the *size* of every effect you report, including the real ones.\n\nYou stop when the estimate is unusually high, because that is what crossing the threshold means. The estimate at that moment is therefore drawn from the upper part of its own distribution. This is the winner's curse, and it is why so many shipped improvements shrink when re-measured: a 4-point gain declared on day three becomes 1.5 points when the experiment is allowed to run out.\n\nIf you have stopped early, say so and treat the effect size as an upper bound until it has been re-measured on new data.\n\n## What to do instead\n\n**Fix the horizon.** Decide the sample size in advance using the power arithmetic from earlier in this block, decide the end date, and look once when it arrives. This is the simplest correct design and it costs nothing but patience.\n\n**If you must look, budget for it.** Group sequential methods spend the error rate across a planned number of looks. O'Brien–Fleming boundaries are strict early and relax towards the end, so an early stop requires overwhelming evidence; Pocock boundaries are constant and stop earlier at the cost of a stricter final threshold. Both require the number and timing of looks to be planned in advance.\n\n**Or use a method built for continuous monitoring.** Sequential tests based on confidence sequences (sometimes marketed as \"always-valid\" p-values) give an interval that is correct at every moment you choose to look. The price is width: the interval is meaningfully wider than a fixed-horizon interval on the same data, which is exactly the cost of the freedom to peek. Several open-source implementations exist, and most mature experimentation platforms now ship one.\n\nThere is a third option nobody names, and it is often the right one: **stop looking**. Most experiments run for a fortnight because a week of traffic is needed, not because anybody will act mid-flight. Hiding the dashboard until the end date removes the temptation entirely, costs nothing, and is easier to enforce than a boundary nobody remembers the rationale for. Where a team insists on visibility, show the sample size accumulating and the guardrails, and withhold the primary metric until the horizon.\n\n## The offline version of the same sin\n\nThis module's arithmetic applies to eval sets too, in a form that looks innocent: run the set, get a non-significant result, add fifty more items, check again, repeat until it turns significant. That is optional stopping with extra steps.\n\nRun the whole set. If it is inconclusive, decide the new size in advance, collect it, and rerun the whole comparison on the enlarged set, reporting that you did so.\n\n## One exception, and it is deliberate\n\nStopping early for harm is not only permitted, it should be planned. Write the guardrail conditions before launch (a safety-violation rate above some level, an error rate above another, a latency regression beyond a stated bound) and stop the moment they trigger, without a significance test.\n\nThe asymmetry is on purpose. Stopping early for a good result buys you a few days and costs you a reliable number. Stopping early for a bad result costs you a few days and protects users. Those are not the same trade, and the stopping rule should not treat them as though they were."},"check":{"question":"An experiment is checked daily and stopped on day four when the improvement first reaches significance at 6.2 points. What is the most likely relationship between 6.2 points and the true effect?","options":[{"text":"It is an unbiased estimate, since the significance test already accounts for how much data was collected.","correct":false,"tempting":"The test corrects for sample size but not for the stopping rule; the decision to stop was itself made on the basis of a high estimate."},{"text":"It understates the true effect, because early data has more noise and noise pulls estimates towards zero.","correct":false,"tempting":"Noise widens the spread in both directions; it does not systematically shrink estimates, and the stopping rule selects from the high side."},{"text":"It overstates the true effect, because stopping was triggered by the estimate being unusually high at that moment.","correct":true},{"text":"It is unrelated to the true effect, since the daily peeking has invalidated the measurement entirely.","correct":false,"tempting":"Too strong: peeking inflates the false-positive rate and biases the estimate upwards, but the data still carries information about a real effect."}]},"prev":{"slug":"many-comparisons","title":"Twenty slices and one guaranteed surprise","url":"https://addaly.com/learn/evaluating-ai/many-comparisons"},"next":{"slug":"variance-reduction","title":"Getting more signal from the items you already have","url":"https://addaly.com/learn/evaluating-ai/variance-reduction"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Watching the dashboard until it says yes', in Knowing If It Works. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/evaluating-ai/peeking-and-stopping-rules (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Watching the dashboard until it says yes","courseTitle":"Knowing If It Works","module":{"index":3,"slug":"deciding-if-it-is-real","title":"Deciding whether a difference is real"},"answer":null,"keyIdea":"Checking an experiment repeatedly and stopping when it crosses the line turns a five per cent error rate into something far larger, and the effect you stop on is overestimated because you stopped at a high point.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"type":"line","title":"The false-positive rate rises with every look","series":[{"label":"Two identical systems, stop at the first p below 0.05","points":[[1,5],[2,8],[3,11],[5,14],[10,18]]}],"xLabel":"Number of peeks at the dashboard","yLabel":"Chance of a false winner, %","caption":"The nominal 5% holds for exactly one look. Ten looks at an experiment with no effect at all declare a winner about one time in five, and the effect reported at the crossing is biased upwards."},"alt":"Figure: The false-positive rate rises with every look. Line chart of Chance of a false winner, % against Number of peeks at the dashboard: Two identical systems, stop at the first p below 0.05 (1, 5) (2, 8) (3, 11) (5, 14) (10, 18). The nominal 5% holds for exactly one look. Ten looks at an experiment with no effect at all declare a winner about one time in five, and the effect reported at the crossing is biased upwards."}},"blocks":[{"t":"h2","id":"what-repeated-looking-does","runs":[{"text":"What repeated looking does"}]},{"t":"p","runs":[{"text":"An online experiment runs for a fortnight. The team checks the dashboard every morning and will stop when the p-value drops below 0.05. This feels efficient and it silently destroys the error rate."}]},{"t":"p","runs":[{"text":"The mechanism is simple. A p-value computed on accumulating data wanders. Even when there is no effect at all, it drifts up and down, and over enough looks it will dip below 0.05 at some point. Checking five times pushes the false-positive rate from 5% to roughly 14%. Checking continuously, with no fixed horizon, means the probability of eventually crossing approaches certainty."}]},{"t":"p","runs":[{"text":"You can watch it happen in a few lines:"}]},{"t":"code","lang":"python","label":"Python","text":"import numpy as np\nrng = np.random.default_rng(0)\nhits = 0\nfor _ in range(2000):\n    a = rng.random(4000) < 0.20        # identical systems\n    b = rng.random(4000) < 0.20\n    for n in range(400, 4001, 400):    # ten peeks\n        pa, pb = a[:n].mean(), b[:n].mean()\n        se = np.sqrt(2 * 0.2 * 0.8 / n)\n        if abs(pb - pa) > 1.96 * se:\n            hits += 1\n            break\nprint(hits / 2000)                      # ≈ 0.18"},{"t":"p","runs":[{"text":"Two identical systems, and eighteen per cent of experiments declare a winner."}]},{"t":"h2","id":"the-second-quieter-damage","runs":[{"text":"The second, quieter damage"}]},{"t":"p","runs":[{"text":"Stopping at the moment of crossing does not only produce false positives. It biases the "},{"text":"size","i":true},{"text":" of every effect you report, including the real ones."}]},{"t":"p","runs":[{"text":"You stop when the estimate is unusually high, because that is what crossing the threshold means. The estimate at that moment is therefore drawn from the upper part of its own distribution. This is the winner’s curse, and it is why so many shipped improvements shrink when re-measured: a 4-point gain declared on day three becomes 1.5 points when the experiment is allowed to run out."}]},{"t":"p","runs":[{"text":"If you have stopped early, say so and treat the effect size as an upper bound until it has been re-measured on new data."}]},{"t":"h2","id":"what-to-do-instead","runs":[{"text":"What to do instead"}]},{"t":"p","runs":[{"text":"Fix the horizon.","b":true},{"text":" Decide the sample size in advance using the power arithmetic from earlier in this block, decide the end date, and look once when it arrives. This is the simplest correct design and it costs nothing but patience."}]},{"t":"p","runs":[{"text":"If you must look, budget for it.","b":true},{"text":" Group sequential methods spend the error rate across a planned number of looks. O’Brien–Fleming boundaries are strict early and relax towards the end, so an early stop requires overwhelming evidence; Pocock boundaries are constant and stop earlier at the cost of a stricter final threshold. Both require the number and timing of looks to be planned in advance."}]},{"t":"p","runs":[{"text":"Or use a method built for continuous monitoring.","b":true},{"text":" Sequential tests based on confidence sequences (sometimes marketed as “always-valid” p-values) give an interval that is correct at every moment you choose to look. The price is width: the interval is meaningfully wider than a fixed-horizon interval on the same data, which is exactly the cost of the freedom to peek. Several open-source implementations exist, and most mature experimentation platforms now ship one."}]},{"t":"p","runs":[{"text":"There is a third option nobody names, and it is often the right one: "},{"text":"stop looking","b":true},{"text":". Most experiments run for a fortnight because a week of traffic is needed, not because anybody will act mid-flight. Hiding the dashboard until the end date removes the temptation entirely, costs nothing, and is easier to enforce than a boundary nobody remembers the rationale for. Where a team insists on visibility, show the sample size accumulating and the guardrails, and withhold the primary metric until the horizon."}]},{"t":"h2","id":"the-offline-version-of-the-same-sin","runs":[{"text":"The offline version of the same sin"}]},{"t":"p","runs":[{"text":"This module’s arithmetic applies to eval sets too, in a form that looks innocent: run the set, get a non-significant result, add fifty more items, check again, repeat until it turns significant. That is optional stopping with extra steps."}]},{"t":"p","runs":[{"text":"Run the whole set. If it is inconclusive, decide the new size in advance, collect it, and rerun the whole comparison on the enlarged set, reporting that you did so."}]},{"t":"h2","id":"one-exception-and-it-is-deliberate","runs":[{"text":"One exception, and it is deliberate"}]},{"t":"p","runs":[{"text":"Stopping early for harm is not only permitted, it should be planned. Write the guardrail conditions before launch (a safety-violation rate above some level, an error rate above another, a latency regression beyond a stated bound) and stop the moment they trigger, without a significance test."}]},{"t":"p","runs":[{"text":"The asymmetry is on purpose. Stopping early for a good result buys you a few days and costs you a reliable number. Stopping early for a bad result costs you a few days and protects users. Those are not the same trade, and the stopping rule should not treat them as though they were."}]}],"check":{"question":[{"text":"An experiment is checked daily and stopped on day four when the improvement first reaches significance at 6.2 points. What is the most likely relationship between 6.2 points and the true effect?"}],"options":[{"text":[{"text":"It is an unbiased estimate, since the significance test already accounts for how much data was collected."}],"correct":false,"tempting":[{"text":"The test corrects for sample size but not for the stopping rule; the decision to stop was itself made on the basis of a high estimate."}]},{"text":[{"text":"It understates the true effect, because early data has more noise and noise pulls estimates towards zero."}],"correct":false,"tempting":[{"text":"Noise widens the spread in both directions; it does not systematically shrink estimates, and the stopping rule selects from the high side."}]},{"text":[{"text":"It overstates the true effect, because stopping was triggered by the estimate being unusually high at that moment."}],"correct":true,"tempting":null},{"text":[{"text":"It is unrelated to the true effect, since the daily peeking has invalidated the measurement entirely."}],"correct":false,"tempting":[{"text":"Too strong: peeking inflates the false-positive rate and biases the estimate upwards, but the data still carries information about a real effect."}]}],"order":[0,1,3,2],"why":null},"prev":{"slug":"many-comparisons","title":"Twenty slices and one guaranteed surprise","minutes":8},"next":{"slug":"variance-reduction","title":"Getting more signal from the items you already have","minutes":9}}}