{"course":{"slug":"evaluating-ai","title":"Knowing If It Works","url":"https://addaly.com/learn/evaluating-ai","level":"intermediate","track":"build"},"lesson":{"slug":"cost-latency-and-drift","title":"Cost, latency, and the model that changed under you","url":"https://addaly.com/learn/evaluating-ai/cost-latency-and-drift","position":68,"of":76,"minutes":9,"answer":"Pin the model and keep a frozen set, and a forced upgrade becomes an afternoon instead of a crisis.","keyIdea":"Pin the model and keep a frozen set, and a forced upgrade becomes an afternoon instead of a crisis."},"body":{"format":"markdown","text":"## Quality is a row, not a number\n\nEvery eval run should print four things together, because they trade against each other and you can only see a trade when the numbers sit side by side.\n\n```\nvariant        pass    p50      p95      cost / 1k calls\nv3 (current)   78%     1.1s     2.4s     $2.10\nv4 (proposed)  81%     2.6s     8.9s     $7.40\nv4-mini        74%     0.6s     1.3s     $0.35\n```\n\nNow the conversation is real. Three points of quality for a p95 that goes from 2.4 to 8.9 seconds is usually a bad trade for anything a person is waiting on. And `v4-mini`, four points worse and six times cheaper, is suddenly the interesting row: at 100,000 calls a day that is the difference between roughly $10,500 and $1,050 a month.\n\nTwo details that matter. Report **p95, not the average**: the average hides the one user in twenty who waits nine seconds, and that user is the one who leaves. And convert cost into your actual monthly bill at your actual volume, in the currency you pay in. \"$0.0074 per call\" and \"about ₹9 lakh a month\" produce completely different reactions in a budget meeting, and only one of them is a decision.\n\n## Pin the model. Always.\n\nNever point production at an alias. A dated, explicit model id in a config file, changed by a pull request that runs the eval. This one habit converts the worst day of your quarter into an ordinary afternoon.\n\nBecause eventually you will be forced to move: a deprecation notice, a price change, a provider outage that pushes you to a fallback. When that day comes, the frozen eval set is the only asset that matters. Run it against both models, read the per-item diff, and you have an answer in an hour rather than a week of arguing in Slack.\n\n## Quality dropped and you changed nothing\n\nA sequence that works.\n\n**Confirm it with the set, not with anecdotes.** Three loud complaints in a channel is not a trend, and sometimes the numbers are flat. Sometimes the numbers confirm it, and now you have a magnitude.\n\n**Look at failures by slice, never in aggregate.** Model changes are almost never uniform. The new model is usually better on the bulk and worse on something specific: non-English input, very long documents, strict JSON with a nested schema, refusals on your borderline-but-legitimate cases. A model can improve overall and get dramatically worse on the 8% of traffic your complainers live in, and the headline number will happily report an improvement.\n\n**Fix the prompt to the new model, do not expect the old prompt to transfer.** Prompts overfit to models. Instructions that were load-bearing for one model are noise to another; formatting tricks that worked stop working. Budget half a day to re-tune, and re-run the set after.\n\n**Keep a canary running.** Sample one percent of live traffic every day through your calibrated judge and chart it. A slow drift is obvious on a chart and invisible in memory. This is also the only thing that catches the other kind of drift, which nobody plans for: *your inputs* changed. New market, new customer segment, a feature that brought in a different sort of question. Your eval set from January describes January.\n\n## The last honest thing\n\nEvaluation is not a phase you complete. It is a small permanent tax: an hour a week keeping the set current, a CI job, a chart someone glances at. Teams that pay it ship changes confidently and recover from surprises in an afternoon. Teams that do not spend that same time arguing about whether things got worse, and finding out from customers when they did."},"check":{"question":"A provider deprecation forces an upgrade. Within days, support complaints about the summariser rise sharply. The team runs its frozen eval set and the pass rate has gone up three points. What is the most useful reading?","options":[{"text":"Both can be true: break the failures down by slice, because a model can improve on the bulk of traffic and get worse on the slice the complainers are in.","correct":true},{"text":"The complaints are anecdotal and the eval is the measurement, so there is nothing to act on.","correct":false,"tempting":"Rightly prefers data to anecdote, but forgets that an eval set only measures the cases someone put into it."},{"text":"The eval set is stale and should be rebuilt from recent traffic before any conclusion is drawn.","correct":false,"tempting":"Has the right instinct about staleness but the wrong order: rebuilding now destroys the before-and-after comparison that could explain the spike."},{"text":"The new model is better and the complaints are a novelty effect that will settle within two weeks.","correct":false,"tempting":"Borrows a concept from interface experiments and applies it to a backend swap users never saw."}]},"prev":{"slug":"contested-quality","title":"When people disagree about what good means","url":"https://addaly.com/learn/evaluating-ai/contested-quality"},"next":{"slug":"latency-is-a-distribution","title":"The tail is what your users feel","url":"https://addaly.com/learn/evaluating-ai/latency-is-a-distribution"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Cost, latency, and the model that changed under you', in Knowing If It Works. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/evaluating-ai/cost-latency-and-drift (Accessed: 31 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Cost, latency, and the model that changed under you","courseTitle":"Knowing If It Works","module":{"index":8,"slug":"living-with-it","title":"Living with it"},"answer":null,"keyIdea":"Pin the model and keep a frozen set, and a forced upgrade becomes an afternoon instead of a crisis.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"quality-is-a-row-not-a-number","runs":[{"text":"Quality is a row, not a number"}]},{"t":"p","runs":[{"text":"Every eval run should print four things together, because they trade against each other and you can only see a trade when the numbers sit side by side."}]},{"t":"code","lang":"","label":"Text","text":"variant        pass    p50      p95      cost / 1k calls\nv3 (current)   78%     1.1s     2.4s     $2.10\nv4 (proposed)  81%     2.6s     8.9s     $7.40\nv4-mini        74%     0.6s     1.3s     $0.35"},{"t":"p","runs":[{"text":"Now the conversation is real. Three points of quality for a p95 that goes from 2.4 to 8.9 seconds is usually a bad trade for anything a person is waiting on. And "},{"text":"v4-mini","code":true},{"text":", four points worse and six times cheaper, is suddenly the interesting row: at 100,000 calls a day that is the difference between roughly $10,500 and $1,050 a month."}]},{"t":"p","runs":[{"text":"Two details that matter. Report "},{"text":"p95, not the average","b":true},{"text":": the average hides the one user in twenty who waits nine seconds, and that user is the one who leaves. And convert cost into your actual monthly bill at your actual volume, in the currency you pay in. “$0.0074 per call” and “about ₹9 lakh a month” produce completely different reactions in a budget meeting, and only one of them is a decision."}]},{"t":"h2","id":"pin-the-model-always","runs":[{"text":"Pin the model. Always."}]},{"t":"p","runs":[{"text":"Never point production at an alias. A dated, explicit model id in a config file, changed by a pull request that runs the eval. This one habit converts the worst day of your quarter into an ordinary afternoon."}]},{"t":"p","runs":[{"text":"Because eventually you will be forced to move: a deprecation notice, a price change, a provider outage that pushes you to a fallback. When that day comes, the frozen eval set is the only asset that matters. Run it against both models, read the per-item diff, and you have an answer in an hour rather than a week of arguing in Slack."}]},{"t":"h2","id":"quality-dropped-and-you-changed-nothing","runs":[{"text":"Quality dropped and you changed nothing"}]},{"t":"p","runs":[{"text":"A sequence that works."}]},{"t":"p","runs":[{"text":"Confirm it with the set, not with anecdotes.","b":true},{"text":" Three loud complaints in a channel is not a trend, and sometimes the numbers are flat. Sometimes the numbers confirm it, and now you have a magnitude."}]},{"t":"p","runs":[{"text":"Look at failures by slice, never in aggregate.","b":true},{"text":" Model changes are almost never uniform. The new model is usually better on the bulk and worse on something specific: non-English input, very long documents, strict JSON with a nested schema, refusals on your borderline-but-legitimate cases. A model can improve overall and get dramatically worse on the 8% of traffic your complainers live in, and the headline number will happily report an improvement."}]},{"t":"p","runs":[{"text":"Fix the prompt to the new model, do not expect the old prompt to transfer.","b":true},{"text":" Prompts overfit to models. Instructions that were load-bearing for one model are noise to another; formatting tricks that worked stop working. Budget half a day to re-tune, and re-run the set after."}]},{"t":"p","runs":[{"text":"Keep a canary running.","b":true},{"text":" Sample one percent of live traffic every day through your calibrated judge and chart it. A slow drift is obvious on a chart and invisible in memory. This is also the only thing that catches the other kind of drift, which nobody plans for: "},{"text":"your inputs","i":true},{"text":" changed. New market, new customer segment, a feature that brought in a different sort of question. Your eval set from January describes January."}]},{"t":"h2","id":"the-last-honest-thing","runs":[{"text":"The last honest thing"}]},{"t":"p","runs":[{"text":"Evaluation is not a phase you complete. It is a small permanent tax: an hour a week keeping the set current, a CI job, a chart someone glances at. Teams that pay it ship changes confidently and recover from surprises in an afternoon. Teams that do not spend that same time arguing about whether things got worse, and finding out from customers when they did."}]}],"check":{"question":[{"text":"A provider deprecation forces an upgrade. Within days, support complaints about the summariser rise sharply. The team runs its frozen eval set and the pass rate has gone up three points. What is the most useful reading?"}],"options":[{"text":[{"text":"Both can be true: break the failures down by slice, because a model can improve on the bulk of traffic and get worse on the slice the complainers are in."}],"correct":true,"tempting":null},{"text":[{"text":"The complaints are anecdotal and the eval is the measurement, so there is nothing to act on."}],"correct":false,"tempting":[{"text":"Rightly prefers data to anecdote, but forgets that an eval set only measures the cases someone put into it."}]},{"text":[{"text":"The eval set is stale and should be rebuilt from recent traffic before any conclusion is drawn."}],"correct":false,"tempting":[{"text":"Has the right instinct about staleness but the wrong order: rebuilding now destroys the before-and-after comparison that could explain the spike."}]},{"text":[{"text":"The new model is better and the complaints are a novelty effect that will settle within two weeks."}],"correct":false,"tempting":[{"text":"Borrows a concept from interface experiments and applies it to a backend swap users never saw."}]}],"order":[3,0,2,1],"why":null},"prev":{"slug":"contested-quality","title":"When people disagree about what good means","minutes":9},"next":{"slug":"latency-is-a-distribution","title":"The tail is what your users feel","minutes":9}}}