Buying a model you will depend on

The comparison that matters takes an afternoon

Take 300 items from your own set, run three candidate models through the same prompt, and measure five things: task success, cost per successful task, p95 latency, format or schema compliance, and refusal rate.

That is an afternoon of work, and it beats every leaderboard, for the mechanical reason established earlier in the course: public benchmarks sit in training corpora, and they describe tasks that are not yours. A model two places lower on a public ranking routinely wins on a specific job because the job rewards instruction-following or JSON discipline rather than reasoning breadth.

Cost per successful task is the number to insist on. A model at half the token price that needs two attempts and produces malformed output on 8% of calls is not cheaper. Compute it as total spend divided by tasks that came out right.

The questions to ask before the integration exists

Answers to these change architecture, and they are much easier to get before you have signed anything.

• Version pinning. Can I pin a dated model version, and for how long is it served? Model behaviour changing under you is the most disruptive thing a supplier can do, and the pinning policy is the mitigation.
• Deprecation notice. How much warning before a version is retired? Thirty days and six months are very different engineering plans.
• Training on your data. Is customer data used for training by default? Can it be disabled, is that in the contract or only the documentation, and does it also cover data sent to any evaluation or safety systems?
• Retention. How long is prompt and output data kept, where, and can retention be shortened? This is often the deciding constraint for a regulated customer.
• Region of processing, and whether it is guaranteed or best-effort.
• Rate limits and behaviour under load, including whether limits are per-key or per-organisation, and what the response looks like when you exceed them.
• Abuse detection. What happens if the automated abuse system flags your traffic (throttling, suspension, a human review) and how do you reach a person quickly.
• Sub-processors and certifications, because your own customers will ask you.

Test the failure modes deliberately

Success paths are easy. Spend an hour on the failures, because they are what your on-call engineer will meet:

• Send an over-length input and record the error shape.
• Exceed the rate limit deliberately and check whether the response carries a retry-after and whether your client honours it.
• Send an input the safety system will refuse, and see whether the refusal is a normal response, an error code, or something your parser cannot handle.
• Kill the connection mid-stream and confirm you can detect a truncated response: the finish reason again.
• Run a hundred requests concurrently and look at the latency tail rather than the mean.

Write the resulting error taxonomy into your integration. Most production incidents with a model supplier are not the model being wrong; they are the client mishandling a shape of failure nobody tested.

Model the cost properly

Token prices are the smallest part of the estimate. Include prompt growth as features are added, the retrieved context you pass, retries, the evaluation and judging traffic, and any cached-input or batch discounts. Then express it per business unit (per resolved ticket, per document processed, per active user) because that is the number that will be compared against the alternative.

Keep the exit cheap

You will change supplier at some point: a price move, a deprecation, an outage, a policy change, a better model. Two practices make that a week rather than a quarter.

Keep provider-specific detail thin. A small adapter layer, prompts that do not depend on one provider’s quirks, and an output parser tolerant of formatting differences.

Re-run your set on a second provider quarterly, and keep the result. A documented fallback that scored within a few points last quarter is an operational option; an untested fallback is a hope. This is also the only way to know whether your prompts have quietly become provider-specific, which they do, gradually, without anybody deciding to.