Cost, latency, and the model that changed under you

Quality is a row, not a number

Every eval run should print four things together, because they trade against each other and you can only see a trade when the numbers sit side by side.

variant        pass    p50      p95      cost / 1k calls
v3 (current)   78%     1.1s     2.4s     $2.10
v4 (proposed)  81%     2.6s     8.9s     $7.40
v4-mini        74%     0.6s     1.3s     $0.35

Now the conversation is real. Three points of quality for a p95 that goes from 2.4 to 8.9 seconds is usually a bad trade for anything a person is waiting on. And v4-mini, four points worse and six times cheaper, is suddenly the interesting row — at 100,000 calls a day that is the difference between roughly $10,500 and $1,050 a month.

Two details that matter. Report p95, not the average: the average hides the one user in twenty who waits nine seconds, and that user is the one who leaves. And convert cost into your actual monthly bill at your actual volume, in the currency you pay in. "$0.0074 per call" and "about ₹9 lakh a month" produce completely different reactions in a budget meeting, and only one of them is a decision.

Pin the model. Always.

Never point production at an alias. A dated, explicit model id in a config file, changed by a pull request that runs the eval. This one habit converts the worst day of your quarter into an ordinary afternoon.

Because eventually you will be forced to move — a deprecation notice, a price change, a provider outage that pushes you to a fallback. When that day comes, the frozen eval set is the only asset that matters. Run it against both models, read the per-item diff, and you have an answer in an hour rather than a week of arguing in Slack.

Quality dropped and you changed nothing

A sequence that works.

Confirm it with the set, not with anecdotes. Three loud complaints in a channel is not a trend, and sometimes the numbers are genuinely flat. Sometimes the numbers confirm it, and now you have a magnitude.

Look at failures by slice, never in aggregate. Model changes are almost never uniform. The new model is usually better on the bulk and worse on something specific: non-English input, very long documents, strict JSON with a nested schema, refusals on your borderline-but-legitimate cases. A model can improve overall and get dramatically worse on the 8% of traffic your complainers live in — and the headline number will happily report an improvement.

Fix the prompt to the new model, do not expect the old prompt to transfer. Prompts overfit to models. Instructions that were load-bearing for one model are noise to another; formatting tricks that worked stop working. Budget half a day to re-tune, and re-run the set after.

Keep a canary running. Sample one percent of live traffic every day through your calibrated judge and chart it. A slow drift is obvious on a chart and invisible in memory. This is also the only thing that catches the other kind of drift, which nobody plans for: *your inputs* changed. New market, new customer segment, a feature that brought in a different sort of question. Your eval set from January describes January.

The last honest thing

Evaluation is not a phase you complete. It is a small permanent tax — an hour a week keeping the set current, a CI job, a chart someone glances at. Teams that pay it ship changes confidently and recover from surprises in an afternoon. Teams that do not spend that same time arguing about whether things got worse, and finding out from customers when they did.