{"course":{"slug":"evaluating-ai","title":"Knowing If It Works","url":"https://addaly.com/learn/evaluating-ai","level":"intermediate","track":"build"},"lesson":{"slug":"evaluating-a-vendor-model","title":"Buying a model you will depend on","url":"https://addaly.com/learn/evaluating-ai/evaluating-a-vendor-model","position":60,"of":76,"minutes":10,"answer":"Decide between suppliers on your own three hundred items and on cost per successful task rather than per token, and get the answers about version pinning, deprecation notice and data retention before the integration exists.","keyIdea":"Decide between suppliers on your own three hundred items and on cost per successful task rather than per token, and get the answers about version pinning, deprecation notice and data retention before the integration exists."},"body":{"format":"markdown","text":"## The comparison that matters takes an afternoon\n\nTake 300 items from your own set, run three candidate models through the same prompt, and measure five things: task success, cost per *successful* task, p95 latency, format or schema compliance, and refusal rate.\n\nThat is an afternoon of work, and it beats every leaderboard, for the mechanical reason established earlier in the course: public benchmarks sit in training corpora, and they describe tasks that are not yours. A model two places lower on a public ranking routinely wins on a specific job because the job rewards instruction-following or JSON discipline rather than reasoning breadth.\n\nCost per successful task is the number to insist on. A model at half the token price that needs two attempts and produces malformed output on 8% of calls is not cheaper. Compute it as total spend divided by tasks that came out right.\n\n> Figure: Price per call is not cost per success. Bars (dollars per 1,000): Model A: price per 1,000 calls 10; Model B: price per 1,000 calls 5; Model A: cost per 1,000 successful tasks 10.6; Model B: cost per 1,000 successful tasks 10.9. Model A succeeds on 94% of calls in one attempt. Model B is half the price, needs two attempts, and still returns malformed output on 8% of calls, so it costs more per task that actually came out right.\n\n## The questions to ask before the integration exists\n\nAnswers to these change architecture, and they are much easier to get before you have signed anything.\n\n- **Version pinning.** Can I pin a dated model version, and for how long is it served? Model behaviour changing under you is the most disruptive thing a supplier can do, and the pinning policy is the mitigation.\n- **Deprecation notice.** How much warning before a version is retired? Thirty days and six months are very different engineering plans.\n- **Training on your data.** Is customer data used for training by default? Can it be disabled, is that in the contract or only the documentation, and does it also cover data sent to any evaluation or safety systems?\n- **Retention.** How long is prompt and output data kept, where, and can retention be shortened? This is often the deciding constraint for a regulated customer.\n- **Region of processing**, and whether it is guaranteed or best-effort.\n- **Rate limits and behaviour under load**, including whether limits are per-key or per-organisation, and what the response looks like when you exceed them.\n- **Abuse detection.** What happens if the automated abuse system flags your traffic (throttling, suspension, a human review) and how do you reach a person quickly.\n- **Sub-processors and certifications**, because your own customers will ask you.\n\n## Test the failure modes deliberately\n\nSuccess paths are easy. Spend an hour on the failures, because they are what your on-call engineer will meet:\n\n- Send an over-length input and record the error shape.\n- Exceed the rate limit deliberately and check whether the response carries a retry-after and whether your client honours it.\n- Send an input the safety system will refuse, and see whether the refusal is a normal response, an error code, or something your parser cannot handle.\n- Kill the connection mid-stream and confirm you can detect a truncated response: the finish reason again.\n- Run a hundred requests concurrently and look at the latency tail rather than the mean.\n\nWrite the resulting error taxonomy into your integration. Most production incidents with a model supplier are not the model being wrong; they are the client mishandling a shape of failure nobody tested.\n\n## Model the cost properly\n\nToken prices are the smallest part of the estimate. Include prompt growth as features are added, the retrieved context you pass, retries, the evaluation and judging traffic, and any cached-input or batch discounts. Then express it per business unit (per resolved ticket, per document processed, per active user) because that is the number that will be compared against the alternative.\n\n## Keep the exit cheap\n\nYou will change supplier at some point: a price move, a deprecation, an outage, a policy change, a better model. Two practices make that a week rather than a quarter.\n\n**Keep provider-specific detail thin.** A small adapter layer, prompts that do not depend on one provider's quirks, and an output parser tolerant of formatting differences.\n\n**Re-run your set on a second provider quarterly**, and keep the result. A documented fallback that scored within a few points last quarter is an operational option; an untested fallback is a hope. This is also the only way to know whether your prompts have quietly become provider-specific, which they do, gradually, without anybody deciding to."},"check":{"question":"Model A costs half as much per token as model B, but produces schema-valid output 92% of the time against B's 99.5%, and each retry costs a full call. What is the sound way to compare them?","options":[{"text":"Compare price per million tokens, then account separately for the engineering time spent handling malformed output.","correct":false,"tempting":"Splits one decision into two incomparable ledgers and leaves the retry traffic out of the price entirely."},{"text":"Compare cost per successful task, including retries and the failures that never succeed.","correct":true},{"text":"Prefer B, since schema compliance below 95% is unacceptable in production regardless of cost.","correct":false,"tempting":"Sets an arbitrary bar in place of the arithmetic; whether 92% is acceptable depends on retry cost and on what a failure does downstream."},{"text":"Compare quality on a public benchmark first, and treat cost as a tie-breaker.","correct":false,"tempting":"Leads with the measurement least connected to the task and defers the one that decides the bill."}]},"prev":{"slug":"writing-a-system-card","title":"The document that travels with the system","url":"https://addaly.com/learn/evaluating-ai/writing-a-system-card"},"next":{"slug":"independent-evaluation","title":"Why the builder's own numbers are weak evidence","url":"https://addaly.com/learn/evaluating-ai/independent-evaluation"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Buying a model you will depend on', in Knowing If It Works. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/evaluating-ai/evaluating-a-vendor-model (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Buying a model you will depend on","courseTitle":"Knowing If It Works","module":{"index":7,"slug":"assurance","title":"Evidence other people can act on"},"answer":null,"keyIdea":"Decide between suppliers on your own three hundred items and on cost per successful task rather than per token, and get the answers about version pinning, deprecation notice and data retention before the integration exists.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"type":"bars","unit":"dollars per 1,000","title":"Price per call is not cost per success","series":[{"label":"Model A: price per 1,000 calls","value":10},{"label":"Model B: price per 1,000 calls","value":5},{"label":"Model A: cost per 1,000 successful tasks","value":10.6},{"label":"Model B: cost per 1,000 successful tasks","value":10.9,"highlight":true}],"caption":"Model A succeeds on 94% of calls in one attempt. Model B is half the price, needs two attempts, and still returns malformed output on 8% of calls, so it costs more per task that actually came out right."},"alt":"Figure: Price per call is not cost per success. Bars (dollars per 1,000): Model A: price per 1,000 calls 10; Model B: price per 1,000 calls 5; Model A: cost per 1,000 successful tasks 10.6; Model B: cost per 1,000 successful tasks 10.9. Model A succeeds on 94% of calls in one attempt. Model B is half the price, needs two attempts, and still returns malformed output on 8% of calls, so it costs more per task that actually came out right."}},"blocks":[{"t":"h2","id":"the-comparison-that-matters-takes-an-afternoon","runs":[{"text":"The comparison that matters takes an afternoon"}]},{"t":"p","runs":[{"text":"Take 300 items from your own set, run three candidate models through the same prompt, and measure five things: task success, cost per "},{"text":"successful","i":true},{"text":" task, p95 latency, format or schema compliance, and refusal rate."}]},{"t":"p","runs":[{"text":"That is an afternoon of work, and it beats every leaderboard, for the mechanical reason established earlier in the course: public benchmarks sit in training corpora, and they describe tasks that are not yours. A model two places lower on a public ranking routinely wins on a specific job because the job rewards instruction-following or JSON discipline rather than reasoning breadth."}]},{"t":"p","runs":[{"text":"Cost per successful task is the number to insist on. A model at half the token price that needs two attempts and produces malformed output on 8% of calls is not cheaper. Compute it as total spend divided by tasks that came out right."}]},{"t":"h2","id":"the-questions-to-ask-before-the-integration-exists","runs":[{"text":"The questions to ask before the integration exists"}]},{"t":"p","runs":[{"text":"Answers to these change architecture, and they are much easier to get before you have signed anything."}]},{"t":"ul","items":[[{"text":"Version pinning.","b":true},{"text":" Can I pin a dated model version, and for how long is it served? Model behaviour changing under you is the most disruptive thing a supplier can do, and the pinning policy is the mitigation."}],[{"text":"Deprecation notice.","b":true},{"text":" How much warning before a version is retired? Thirty days and six months are very different engineering plans."}],[{"text":"Training on your data.","b":true},{"text":" Is customer data used for training by default? Can it be disabled, is that in the contract or only the documentation, and does it also cover data sent to any evaluation or safety systems?"}],[{"text":"Retention.","b":true},{"text":" How long is prompt and output data kept, where, and can retention be shortened? This is often the deciding constraint for a regulated customer."}],[{"text":"Region of processing","b":true},{"text":", and whether it is guaranteed or best-effort."}],[{"text":"Rate limits and behaviour under load","b":true},{"text":", including whether limits are per-key or per-organisation, and what the response looks like when you exceed them."}],[{"text":"Abuse detection.","b":true},{"text":" What happens if the automated abuse system flags your traffic (throttling, suspension, a human review) and how do you reach a person quickly."}],[{"text":"Sub-processors and certifications","b":true},{"text":", because your own customers will ask you."}]]},{"t":"h2","id":"test-the-failure-modes-deliberately","runs":[{"text":"Test the failure modes deliberately"}]},{"t":"p","runs":[{"text":"Success paths are easy. Spend an hour on the failures, because they are what your on-call engineer will meet:"}]},{"t":"ul","items":[[{"text":"Send an over-length input and record the error shape."}],[{"text":"Exceed the rate limit deliberately and check whether the response carries a retry-after and whether your client honours it."}],[{"text":"Send an input the safety system will refuse, and see whether the refusal is a normal response, an error code, or something your parser cannot handle."}],[{"text":"Kill the connection mid-stream and confirm you can detect a truncated response: the finish reason again."}],[{"text":"Run a hundred requests concurrently and look at the latency tail rather than the mean."}]]},{"t":"p","runs":[{"text":"Write the resulting error taxonomy into your integration. Most production incidents with a model supplier are not the model being wrong; they are the client mishandling a shape of failure nobody tested."}]},{"t":"h2","id":"model-the-cost-properly","runs":[{"text":"Model the cost properly"}]},{"t":"p","runs":[{"text":"Token prices are the smallest part of the estimate. Include prompt growth as features are added, the retrieved context you pass, retries, the evaluation and judging traffic, and any cached-input or batch discounts. Then express it per business unit (per resolved ticket, per document processed, per active user) because that is the number that will be compared against the alternative."}]},{"t":"h2","id":"keep-the-exit-cheap","runs":[{"text":"Keep the exit cheap"}]},{"t":"p","runs":[{"text":"You will change supplier at some point: a price move, a deprecation, an outage, a policy change, a better model. Two practices make that a week rather than a quarter."}]},{"t":"p","runs":[{"text":"Keep provider-specific detail thin.","b":true},{"text":" A small adapter layer, prompts that do not depend on one provider’s quirks, and an output parser tolerant of formatting differences."}]},{"t":"p","runs":[{"text":"Re-run your set on a second provider quarterly","b":true},{"text":", and keep the result. A documented fallback that scored within a few points last quarter is an operational option; an untested fallback is a hope. This is also the only way to know whether your prompts have quietly become provider-specific, which they do, gradually, without anybody deciding to."}]}],"check":{"question":[{"text":"Model A costs half as much per token as model B, but produces schema-valid output 92% of the time against B’s 99.5%, and each retry costs a full call. What is the sound way to compare them?"}],"options":[{"text":[{"text":"Compare price per million tokens, then account separately for the engineering time spent handling malformed output."}],"correct":false,"tempting":[{"text":"Splits one decision into two incomparable ledgers and leaves the retry traffic out of the price entirely."}]},{"text":[{"text":"Compare cost per successful task, including retries and the failures that never succeed."}],"correct":true,"tempting":null},{"text":[{"text":"Prefer B, since schema compliance below 95% is unacceptable in production regardless of cost."}],"correct":false,"tempting":[{"text":"Sets an arbitrary bar in place of the arithmetic; whether 92% is acceptable depends on retry cost and on what a failure does downstream."}]},{"text":[{"text":"Compare quality on a public benchmark first, and treat cost as a tie-breaker."}],"correct":false,"tempting":[{"text":"Leads with the measurement least connected to the task and defers the one that decides the bill."}]}],"order":[2,0,1,3],"why":null},"prev":{"slug":"writing-a-system-card","title":"The document that travels with the system","minutes":9},"next":{"slug":"independent-evaluation","title":"Why the builder’s own numbers are weak evidence","minutes":8}}}