Getting evidence from live traffic without exposing users

Three stages between the eval set and everyone

An offline set says the change is probably good. Live traffic says whether it is. The gap between them is crossed in three steps, each answering a different question.

Shadow: the new system processes real inputs and its outputs are discarded. Answers: does it behave sanely on the real distribution.

Canary: a small share of real users get the new system. Answers: does anything break at volume, and do the guardrails hold.

Staged ramp: the share grows on a schedule with soak time. Answers: does it survive a full day, a Monday, a month-end.

Shadow mode, and the incident everybody has once

Shadow runs are cheap evidence and they have one sharp edge: side effects must be disabled. The new agent must not send the email, write the record, call the payment API or post the message. Every team that skips this learns it in the same way, and the second copy of a customer email is the polite version of the lesson.

Enforce it structurally, not by convention: a shadow request carries a flag that the tool layer checks, and effectful tools return a simulated result. Then verify it, by running the shadow path against a staging tool endpoint that records calls and asserts none arrived.

What shadow gives you, with no labels at all:

• Distribution comparisons on real inputs: output length, refusal rate, groundedness rate, schema validity, tool-call frequency, cost, latency percentiles. A shift in any of these on real traffic is a real finding.
• A paired comparison on thousands of items, since old and new ran on identical inputs. Every label-free check from earlier in the course applies directly.
• Error discovery: the inputs that crash the new path, which your set never contained.

It cannot tell you whether users prefer the result, and it doubles inference cost while it runs. Run it for a day or two on a sample, not indefinitely on everything.

Canary, sized by the guardrail

The share is chosen by arithmetic, not by feel: how many events do you need before the guardrail metric could plausibly trigger. If your guardrail is a safety-violation rate that occurs in one request per thousand, a 1% canary on 50,000 daily requests gives you 500 requests a day and roughly one expected event: far too slow. Either raise the share or pick a guardrail with a higher base rate.

Two practical rules. Keep assignment sticky per user, or somebody gets two different behaviours in one session and reports a bug that is your experiment. And check whether the canary population is representative: a canary routed to one region, one platform or one data centre is a biased sample, and the mix-shift arithmetic from the statistics block applies exactly.

Guardrails, thresholds and a rehearsed rollback

Write these before launch, with numbers:

• The metrics that can stop the rollout: error rate, safety violations, refusal rate, p95 latency, cost per request, a business guardrail such as escalation rate.
• The threshold and the window for each, sized so that normal fluctuation does not trigger them daily.
• Who is paged, and what they are authorised to do without a meeting.

Then rehearse the rollback. A rollback that has never been executed is a hypothesis, and the moment you discover the config takes ten minutes to propagate should not be during an incident. Roll back once on purpose, in the middle of a quiet afternoon, and time it.

The ramp, and why soak time matters

1%, 5%, 25%, 100%, with a minimum soak at each step: usually a full business day, and at least one weekly cycle before the final step. Some failures only appear with time or volume: rate limits at peak, cache eviction under load, a nightly batch that now takes longer than its window, a weekly report built on a field the new version stopped writing.

The limit worth stating: this machinery detects fast, visible harms. It does not detect slow ones: users gradually trusting an answer that is subtly wrong, a corpus filling with the system’s own output, a group of users quietly giving up. Those are the subject of the final block, and no ramp schedule substitutes for them.