Watching the dashboard until it says yes

What repeated looking does

An online experiment runs for a fortnight. The team checks the dashboard every morning and will stop when the p-value drops below 0.05. This feels efficient and it silently destroys the error rate.

The mechanism is simple. A p-value computed on accumulating data wanders. Even when there is no effect at all, it drifts up and down, and over enough looks it will dip below 0.05 at some point. Checking five times pushes the false-positive rate from 5% to roughly 14%. Checking continuously, with no fixed horizon, means the probability of eventually crossing approaches certainty.

You can watch it happen in a few lines:

import numpy as np
rng = np.random.default_rng(0)
hits = 0
for _ in range(2000):
    a = rng.random(4000) < 0.20        # identical systems
    b = rng.random(4000) < 0.20
    for n in range(400, 4001, 400):    # ten peeks
        pa, pb = a[:n].mean(), b[:n].mean()
        se = np.sqrt(2 * 0.2 * 0.8 / n)
        if abs(pb - pa) > 1.96 * se:
            hits += 1
            break
print(hits / 2000)                      # ≈ 0.18

Two identical systems, and eighteen per cent of experiments declare a winner.

The second, quieter damage

Stopping at the moment of crossing does not only produce false positives. It biases the size of every effect you report, including the real ones.

You stop when the estimate is unusually high, because that is what crossing the threshold means. The estimate at that moment is therefore drawn from the upper part of its own distribution. This is the winner’s curse, and it is why so many shipped improvements shrink when re-measured: a 4-point gain declared on day three becomes 1.5 points when the experiment is allowed to run out.

If you have stopped early, say so and treat the effect size as an upper bound until it has been re-measured on new data.

What to do instead

Fix the horizon. Decide the sample size in advance using the power arithmetic from earlier in this block, decide the end date, and look once when it arrives. This is the simplest correct design and it costs nothing but patience.

If you must look, budget for it. Group sequential methods spend the error rate across a planned number of looks. O’Brien–Fleming boundaries are strict early and relax towards the end, so an early stop requires overwhelming evidence; Pocock boundaries are constant and stop earlier at the cost of a stricter final threshold. Both require the number and timing of looks to be planned in advance.

Or use a method built for continuous monitoring. Sequential tests based on confidence sequences (sometimes marketed as “always-valid” p-values) give an interval that is correct at every moment you choose to look. The price is width: the interval is meaningfully wider than a fixed-horizon interval on the same data, which is exactly the cost of the freedom to peek. Several open-source implementations exist, and most mature experimentation platforms now ship one.

There is a third option nobody names, and it is often the right one: stop looking. Most experiments run for a fortnight because a week of traffic is needed, not because anybody will act mid-flight. Hiding the dashboard until the end date removes the temptation entirely, costs nothing, and is easier to enforce than a boundary nobody remembers the rationale for. Where a team insists on visibility, show the sample size accumulating and the guardrails, and withhold the primary metric until the horizon.

The offline version of the same sin

This module’s arithmetic applies to eval sets too, in a form that looks innocent: run the set, get a non-significant result, add fifty more items, check again, repeat until it turns significant. That is optional stopping with extra steps.

Run the whole set. If it is inconclusive, decide the new size in advance, collect it, and rerun the whole comparison on the enlarged set, reporting that you did so.

One exception, and it is deliberate

Stopping early for harm is not only permitted, it should be planned. Write the guardrail conditions before launch (a safety-violation rate above some level, an error rate above another, a latency regression beyond a stated bound) and stop the moment they trigger, without a significance test.

The asymmetry is on purpose. Stopping early for a good result buys you a few days and costs you a reliable number. Stopping early for a bad result costs you a few days and protects users. Those are not the same trade, and the stopping rule should not treat them as though they were.