When the API simply wins, and when it cannot

The lesson this course owes you

A course on running models yourself has an obvious incentive to conclude that you should. This lesson exists to say where that conclusion is wrong, in enough detail to be useful, because a reader who self-hosts something that should have been an API call will blame the hardware for a decision the reader made.

Where hosted wins

Frontier capability. The best hosted models are still ahead of the best open weights on the hardest reasoning, the longest contexts and the most difficult code, and the gap reopens with each release cycle even as it narrows between them. If your task sits near the limit of what any model can do, the open weights you can run at home will not do it, and no configuration changes that.

Low and spiky volume. Below the break-even volume, hosted is cheaper, and far below it hosted is cheaper by a wide margin. A demand curve with a peak ten times its average is exactly what elastic hosted capacity is for and exactly what one card is bad at.

No operations. Nothing to patch, no driver matrix, no unit file, no one to be on call. For a two-person company this is frequently decisive on its own.

Pace. A better model arrives and you change one string. Locally, a better model arrives and you audition it, re-quantise it, re-test the template and re-tune the serving parameters.

Breadth. Long context that actually works, vision, audio, tool ecosystems, structured output that is well tested. Open equivalents exist for most of these, and assembling them is work you would be doing yourself.

Where local wins in ways no price can settle

Data that cannot leave. A contractual or statutory restriction is not a preference to be traded against a token price. Note that this argument requires the machine to be under your control: rented hardware puts you back in the same category of decision as a hosted API.

Offline and unreliable connectivity. A model on the machine works on a train, in a factory, in a field, and during an outage. This is a capability difference rather than a cost difference.

Latency at small scale. A small local model answers a classification in tens of milliseconds with no network round trip. For a keystroke-latency feature, local is not cheaper; it is the only thing fast enough.

Predictable cost. A fixed monthly figure is easier to budget and, in many organisations, easier to approve than a metered bill that can be surprising. This is an organisational reality and a perfectly good reason.

Access. For a substantial part of this course’s readership, a hosted API requires an international payment card, a billing address a provider accepts, and connectivity good enough to depend on. Where any of those is missing, the comparison is not between two prices. It is between a model that runs and no model at all, and a used card plus a 4-bit quantisation is the entire path.

Freedom to experiment. No rate limit, no content policy shaping your research, no deprecation of the version you validated against, no telemetry. For anyone evaluating models, studying their behaviour or building on a frozen artefact, this is the whole argument.

The decision in one paragraph

Start hosted unless something on the local list applies. Measure your real volumes and your real task difficulty for a month. If volume crosses the break-even and the task is comfortably inside what open weights do well, move the high-volume, low-difficulty work local and keep the hard work hosted. That is the hybrid shape from the previous module, and it is where most sustainable deployments land.

The reason this ordering works is asymmetric risk. Starting hosted and moving local later costs you a migration. Starting local and discovering the weights cannot do the job costs you the hardware, the hours, and the credibility of whoever proposed it.

What to re-examine, and when

Set a date rather than a feeling. Every six months, re-run the audition with the current best open model in your size class and re-run the break-even with your measured volumes. Both sides move: open weights improve, and hosted prices fall. A decision that was right last year is not therefore right now, and neither is the opposite.

The question is never whether local is better. It is which of your workloads is on which side of a line you have measured.