Local first, API behind it

8 min read

Figure

The argument that stops being a choice

Most of this course has compared local against hosted as if you had to pick. In deployments that survive, people usually do not. They run a small local model for the work it handles well and route the rest to an API, and the interesting engineering is in the routing rule rather than in either endpoint.

This is worth taking seriously because it fixes the two things that most often kill a purely local deployment: the capability ceiling the weights impose, and the fact that one machine is one machine and it will be down at some point.

Three patterns, with different properties

Fallback. Try local; on failure (timeout, out of memory, service down) repeat the call against a hosted model. Simple, and it makes a single machine survivable. The property to be careful about is that failure is when your most sensitive request is most likely to go somewhere else, which is exactly backwards if the reason for local was data residency. Fallback needs an allow-list: requests flagged as sensitive fail rather than travel.

Cascade. Run local first, judge the answer, escalate when it is not good enough. The judgement is the hard part. A confidence score from the model itself is unreliable. What works in practice is a cheap deterministic check tied to the task: does the JSON parse, does the SQL run against an empty schema, does the answer cite a retrieved document, is the classification one of the permitted labels. Those are all free and all better than asking the model how sure it is.

Split by task. Decide up front which work each endpoint does. Classification, extraction, routing, short rewrites and embeddings go local, where a 7B model at 4-bit is adequate and the volume is high. Long-horizon reasoning, difficult code, and anything where a mistake is expensive goes to the API. This needs no runtime judgement, is trivially explainable to a colleague, and captures most of the cost saving because the high-volume tasks are the cheap ones.

The gateway makes this one line

Because every engine and every major provider speaks the same request shape, the routing lives in configuration rather than in application code:

YAML

model_list:
  - model_name: classify            # high volume, local
    litellm_params: { model: openai/qwen2.5-7b-instruct, api_base: http://127.0.0.1:8000/v1 }
  - model_name: reason              # low volume, hosted
    litellm_params: { model: <hosted-model-id> }
router_settings:
  fallbacks: [{ classify: ["reason"] }]
  num_retries: 1
  timeout: 30

Applications ask for classify or reason. You change what those mean without touching a line of their code, which also means you can move a task back to local once a better open model lands, or move it out when local quality is not holding.

Measure the split, because it is the whole business case

The number that decides whether this was worth building is the deflection rate: the fraction of requests served locally without escalation. Instrument it from the start, because a cascade that escalates 70% of the time costs more than going straight to the API: you paid for the local pass, paid for the check, and then paid the API anyway.

Text

blended cost = local_share x local_cost + (1 - local_share) x api_cost + escalation_overhead

A deflection rate above roughly 80% on your highest-volume task is where this clearly pays. Below 50% on that task, simplify: go straight to the API and keep local for the specific work where privacy or latency forces it.

What gets worse

You now have two sets of behaviour to reason about. The same prompt produces different output depending on which path it took, so a bug report needs to record the path. Sampling defaults differ between engines and providers, so “it changed” may mean only that it went the other way. Costs arrive in two forms, one metered and one sunk. And every request that crosses to the API crosses whatever boundary you were trying to keep, which is the single point to be explicit about with whoever asked for local in the first place.

Write the rule down in one sentence and put it where users can read it: “Requests tagged internal-only are served locally or refused; everything else may be sent to a third-party provider.” A hybrid system whose routing rule is undocumented is a data-residency claim nobody can verify.

Hybrid is not a compromise between local and hosted. It is a decision about which requests are allowed to leave, expressed in code.

Every lesson here is open. Your progress is recorded once you pass the Open Model Ecosystem exam.

A cascade sends every request to a local 7B model first and escalates to a hosted model whenever the local answer looks weak. Measured over a month, 70% of requests escalate. What follows?

Pick the one you would defend. Your answer stays private.