When the model is right and the product still fails

The gap that surprises everybody once

A support assistant measures 90% correct on a careful evaluation set. In production, 40% of the people who start a question end up with their problem solved.

Nothing is wrong with the measurement. The other 50 points went somewhere else:

• People did not know what they could ask, so they asked something the system does not cover.
• The answer was correct and 300 words long, and nobody read past the first line.
• It took nine seconds, and a fifth of people left before it arrived.
• It was right but not actionable: correct information, no next step, no link, no button.
• It was right and the user did not believe it, because nothing showed where it came from.

Model accuracy is a component measurement. The product measurement is a funnel.

Measure the funnel

Instrument the stages and report them together:

started a question        100%
got a response             94%   (6% abandoned or errored)
response was correct       85%   (of those, by sampled review)
user acted on it           58%
problem resolved           41%   (no follow-up contact within 7 days)

Every step between two rows is a fixable product problem, and the arithmetic tells you which one is worth a week. In this shape the largest loss is between “correct” and “acted on”, which is a writing and interface problem, not a model problem, and no amount of prompt work would have found it.

The resolution measure at the bottom matters most and is usually available already: a repeat contact within a week, a completed transaction, a claim submitted. Use an existing product event rather than inventing a satisfaction survey nobody answers.

Interface decisions change the numbers more than model changes

Three that consistently move the funnel:

• Show the source beside the answer. It raises trust where the answer is right and, more usefully, lets the user catch it when it is wrong. This is the detectability axis from the harm lesson, purchased with a link.
• Draft rather than send. Anywhere the system acts, an editable draft converts an irreversible error into a normal edit. Measure the edit rate; it is also free quality data.
• Be shorter. People do not read. A correct 30-word answer beats a correct 300-word answer at every stage of the funnel after “was correct”, and the eval set cannot see this because a rubric scores completeness.

Displaying confidence deserves a caution: an uncalibrated confidence number is worse than none, because it moves people’s trust with no information behind it. Calibrate first, or show provenance instead.

Attribute the movement, do not just watch it

Product metrics move for reasons that have nothing to do with your model: a campaign, a holiday, an app release, a competitor’s outage, a change to the help centre. So a funnel on its own cannot tell you whether the model change helped.

The pairing that works is the offline set for attribution and the funnel for consequence. When the offline number is flat and resolution jumped, look outside the model. When the offline number rose and resolution did not, look at the stages after “was correct”. Neither measurement answers the question alone, and together they usually name the stage to work on within an afternoon.

Watch five people use it

Five to eight moderated sessions, an hour each, will tell you things no metric will. The recurring findings are consistent across products: people do not know what to type; they do not read; they distrust anything with no source; and they will not scroll.

This is not statistics and it is not meant to be. It is the fastest available route to knowing which part of the funnel to instrument next.

The fallback is part of the product

Every AI feature has a bad day: the provider is down, the model refuses, the request times out, the answer is empty. What the user sees then is part of the measured product, and it is usually built last and never evaluated.

Measure the fallback path explicitly: how often it fires, and what proportion of users reach a resolution through it. A graceful fallback that hands over the help article and a contact route can carry a surprising share of your traffic without anybody noticing, which is the point.