{"course":{"slug":"running-models-yourself","title":"Running Models Yourself","url":"https://addaly.com/learn/running-models-yourself","level":"intermediate","track":"deep"},"lesson":{"slug":"hybrid-local-and-api","title":"Local first, API behind it","url":"https://addaly.com/learn/running-models-yourself/hybrid-local-and-api","position":64,"of":82,"minutes":8,"answer":"Running local first with an API behind it works when the routing rule is explicit: split by task for the high-volume cheap work, escalate on a deterministic check rather than the model's own confidence.","keyIdea":"Running local first with an API behind it works when the routing rule is explicit: split by task for the high-volume cheap work, escalate on a deterministic check rather than the model's own confidence, and measure the deflection rate, because a cascade that escalates most of the time costs more than not having one."},"body":{"format":"markdown","text":"## The argument that stops being a choice\n\nMost of this course has compared local against hosted as if you had to pick. In deployments that survive, people usually do not. They run a small local model for the work it handles well and route the rest to an API, and the interesting engineering is in the routing rule rather than in either endpoint.\n\nThis is worth taking seriously because it fixes the two things that most often kill a purely local deployment: the capability ceiling the weights impose, and the fact that one machine is one machine and it will be down at some point.\n\n## Three patterns, with different properties\n\n**Fallback.** Try local; on failure (timeout, out of memory, service down) repeat the call against a hosted model. Simple, and it makes a single machine survivable. The property to be careful about is that failure is when your most sensitive request is most likely to go somewhere else, which is exactly backwards if the reason for local was data residency. Fallback needs an allow-list: requests flagged as sensitive fail rather than travel.\n\n**Cascade.** Run local first, judge the answer, escalate when it is not good enough. The judgement is the hard part. A confidence score from the model itself is unreliable. What works in practice is a cheap deterministic check tied to the task: does the JSON parse, does the SQL run against an empty schema, does the answer cite a retrieved document, is the classification one of the permitted labels. Those are all free and all better than asking the model how sure it is.\n\n**Split by task.** Decide up front which work each endpoint does. Classification, extraction, routing, short rewrites and embeddings go local, where a 7B model at 4-bit is adequate and the volume is high. Long-horizon reasoning, difficult code, and anything where a mistake is expensive goes to the API. This needs no runtime judgement, is trivially explainable to a colleague, and captures most of the cost saving because the high-volume tasks are the cheap ones.\n\n## The gateway makes this one line\n\nBecause every engine and every major provider speaks the same request shape, the routing lives in configuration rather than in application code:\n\n```yaml\nmodel_list:\n  - model_name: classify            # high volume, local\n    litellm_params: { model: openai/qwen2.5-7b-instruct, api_base: http://127.0.0.1:8000/v1 }\n  - model_name: reason              # low volume, hosted\n    litellm_params: { model: <hosted-model-id> }\nrouter_settings:\n  fallbacks: [{ classify: [\"reason\"] }]\n  num_retries: 1\n  timeout: 30\n```\n\nApplications ask for `classify` or `reason`. You change what those mean without touching a line of their code, which also means you can move a task back to local once a better open model lands, or move it out when local quality is not holding.\n\n> Figure: A cascade only pays at a high deflection rate. Bars (blended cost per million tokens, in cents): Straight to the API, no local pass 50; Cascade, 20 per cent served locally 43; Cascade, 50 per cent served locally 30; Cascade, 80 per cent served locally 16; Cascade, 95 per cent served locally 9. Local share times local cost, plus the rest at the hosted price, plus the escalation overhead you paid on every request that escalated anyway. At fifty cents a million hosted, five locally and two of overhead, a cascade deflecting a fifth of the traffic has bought almost nothing for a great deal of machinery.\n\n## Measure the split, because it is the whole business case\n\nThe number that decides whether this was worth building is the **deflection rate**: the fraction of requests served locally without escalation. Instrument it from the start, because a cascade that escalates 70% of the time costs more than going straight to the API: you paid for the local pass, paid for the check, and then paid the API anyway.\n\n```\nblended cost = local_share x local_cost + (1 - local_share) x api_cost + escalation_overhead\n```\n\nA deflection rate above roughly 80% on your highest-volume task is where this clearly pays. Below 50% on that task, simplify: go straight to the API and keep local for the specific work where privacy or latency forces it.\n\n## What gets worse\n\nYou now have two sets of behaviour to reason about. The same prompt produces different output depending on which path it took, so a bug report needs to record the path. Sampling defaults differ between engines and providers, so \"it changed\" may mean only that it went the other way. Costs arrive in two forms, one metered and one sunk. And every request that crosses to the API crosses whatever boundary you were trying to keep, which is the single point to be explicit about with whoever asked for local in the first place.\n\nWrite the rule down in one sentence and put it where users can read it: \"Requests tagged `internal-only` are served locally or refused; everything else may be sent to a third-party provider.\" A hybrid system whose routing rule is undocumented is a data-residency claim nobody can verify.\n\n> Hybrid is not a compromise between local and hosted. It is a decision about which requests are allowed to leave, expressed in code."},"check":{"question":"A cascade sends every request to a local 7B model first and escalates to a hosted model whenever the local answer looks weak. Measured over a month, 70% of requests escalate. What follows?","options":[{"text":"The local model should be quantised less aggressively, since escalation at this rate indicates quality loss from quantisation.","correct":false,"tempting":"Names one possible cause of weak answers without addressing the economics; even a perfect local pass is wasted if most work still goes out."},{"text":"The escalation threshold is too strict and should be loosened until the deflection rate exceeds 80%.","correct":false,"tempting":"Optimises the metric rather than the outcome, by pushing through local answers that the check has already judged inadequate."},{"text":"The hosted model should be placed first with the local model as fallback, reversing the order.","correct":false,"tempting":"Reverses the order without changing the arithmetic, and gives up the privacy and latency reasons for trying local first."},{"text":"It costs more than calling the hosted model directly: every escalated request paid twice before paying the API.","correct":true}]},"prev":{"slug":"sharing-one-gpu-fairly","title":"One card, several people","url":"https://addaly.com/learn/running-models-yourself/sharing-one-gpu-fairly"},"next":{"slug":"the-honest-cost-comparison","title":"The honest cost comparison","url":"https://addaly.com/learn/running-models-yourself/the-honest-cost-comparison"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Local first, API behind it', in Running Models Yourself. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/running-models-yourself/hybrid-local-and-api (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Local first, API behind it","courseTitle":"Running Models Yourself","module":{"index":7,"slug":"serving-it-to-others","title":"Serving it to other people"},"answer":null,"keyIdea":"Running local first with an API behind it works when the routing rule is explicit: split by task for the high-volume cheap work, escalate on a deterministic check rather than the model’s own confidence, and measure the deflection rate, because a cascade that escalates most of the time costs more than not having one.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"type":"bars","unit":"blended cost per million tokens, in cents","title":"A cascade only pays at a high deflection rate","series":[{"label":"Straight to the API, no local pass","value":50},{"label":"Cascade, 20 per cent served locally","value":43},{"label":"Cascade, 50 per cent served locally","value":30},{"label":"Cascade, 80 per cent served locally","value":16,"highlight":true},{"label":"Cascade, 95 per cent served locally","value":9}],"caption":"Local share times local cost, plus the rest at the hosted price, plus the escalation overhead you paid on every request that escalated anyway. At fifty cents a million hosted, five locally and two of overhead, a cascade deflecting a fifth of the traffic has bought almost nothing for a great deal of machinery."},"alt":"Figure: A cascade only pays at a high deflection rate. Bars (blended cost per million tokens, in cents): Straight to the API, no local pass 50; Cascade, 20 per cent served locally 43; Cascade, 50 per cent served locally 30; Cascade, 80 per cent served locally 16; Cascade, 95 per cent served locally 9. Local share times local cost, plus the rest at the hosted price, plus the escalation overhead you paid on every request that escalated anyway. At fifty cents a million hosted, five locally and two of overhead, a cascade deflecting a fifth of the traffic has bought almost nothing for a great deal of machinery."}},"blocks":[{"t":"h2","id":"the-argument-that-stops-being-a-choice","runs":[{"text":"The argument that stops being a choice"}]},{"t":"p","runs":[{"text":"Most of this course has compared local against hosted as if you had to pick. In deployments that survive, people usually do not. They run a small local model for the work it handles well and route the rest to an API, and the interesting engineering is in the routing rule rather than in either endpoint."}]},{"t":"p","runs":[{"text":"This is worth taking seriously because it fixes the two things that most often kill a purely local deployment: the capability ceiling the weights impose, and the fact that one machine is one machine and it will be down at some point."}]},{"t":"h2","id":"three-patterns-with-different-properties","runs":[{"text":"Three patterns, with different properties"}]},{"t":"p","runs":[{"text":"Fallback.","b":true},{"text":" Try local; on failure (timeout, out of memory, service down) repeat the call against a hosted model. Simple, and it makes a single machine survivable. The property to be careful about is that failure is when your most sensitive request is most likely to go somewhere else, which is exactly backwards if the reason for local was data residency. Fallback needs an allow-list: requests flagged as sensitive fail rather than travel."}]},{"t":"p","runs":[{"text":"Cascade.","b":true},{"text":" Run local first, judge the answer, escalate when it is not good enough. The judgement is the hard part. A confidence score from the model itself is unreliable. What works in practice is a cheap deterministic check tied to the task: does the JSON parse, does the SQL run against an empty schema, does the answer cite a retrieved document, is the classification one of the permitted labels. Those are all free and all better than asking the model how sure it is."}]},{"t":"p","runs":[{"text":"Split by task.","b":true},{"text":" Decide up front which work each endpoint does. Classification, extraction, routing, short rewrites and embeddings go local, where a 7B model at 4-bit is adequate and the volume is high. Long-horizon reasoning, difficult code, and anything where a mistake is expensive goes to the API. This needs no runtime judgement, is trivially explainable to a colleague, and captures most of the cost saving because the high-volume tasks are the cheap ones."}]},{"t":"h2","id":"the-gateway-makes-this-one-line","runs":[{"text":"The gateway makes this one line"}]},{"t":"p","runs":[{"text":"Because every engine and every major provider speaks the same request shape, the routing lives in configuration rather than in application code:"}]},{"t":"code","lang":"yaml","label":"YAML","text":"model_list:\n  - model_name: classify            # high volume, local\n    litellm_params: { model: openai/qwen2.5-7b-instruct, api_base: http://127.0.0.1:8000/v1 }\n  - model_name: reason              # low volume, hosted\n    litellm_params: { model: <hosted-model-id> }\nrouter_settings:\n  fallbacks: [{ classify: [\"reason\"] }]\n  num_retries: 1\n  timeout: 30"},{"t":"p","runs":[{"text":"Applications ask for "},{"text":"classify","code":true},{"text":" or "},{"text":"reason","code":true},{"text":". You change what those mean without touching a line of their code, which also means you can move a task back to local once a better open model lands, or move it out when local quality is not holding."}]},{"t":"h2","id":"measure-the-split-because-it-is-the-whole-business-case","runs":[{"text":"Measure the split, because it is the whole business case"}]},{"t":"p","runs":[{"text":"The number that decides whether this was worth building is the "},{"text":"deflection rate","b":true},{"text":": the fraction of requests served locally without escalation. Instrument it from the start, because a cascade that escalates 70% of the time costs more than going straight to the API: you paid for the local pass, paid for the check, and then paid the API anyway."}]},{"t":"code","lang":"","label":"Text","text":"blended cost = local_share x local_cost + (1 - local_share) x api_cost + escalation_overhead"},{"t":"p","runs":[{"text":"A deflection rate above roughly 80% on your highest-volume task is where this clearly pays. Below 50% on that task, simplify: go straight to the API and keep local for the specific work where privacy or latency forces it."}]},{"t":"h2","id":"what-gets-worse","runs":[{"text":"What gets worse"}]},{"t":"p","runs":[{"text":"You now have two sets of behaviour to reason about. The same prompt produces different output depending on which path it took, so a bug report needs to record the path. Sampling defaults differ between engines and providers, so “it changed” may mean only that it went the other way. Costs arrive in two forms, one metered and one sunk. And every request that crosses to the API crosses whatever boundary you were trying to keep, which is the single point to be explicit about with whoever asked for local in the first place."}]},{"t":"p","runs":[{"text":"Write the rule down in one sentence and put it where users can read it: “Requests tagged "},{"text":"internal-only","code":true},{"text":" are served locally or refused; everything else may be sent to a third-party provider.” A hybrid system whose routing rule is undocumented is a data-residency claim nobody can verify."}]},{"t":"quote","runs":[{"text":"Hybrid is not a compromise between local and hosted. It is a decision about which requests are allowed to leave, expressed in code."}]}],"check":{"question":[{"text":"A cascade sends every request to a local 7B model first and escalates to a hosted model whenever the local answer looks weak. Measured over a month, 70% of requests escalate. What follows?"}],"options":[{"text":[{"text":"The local model should be quantised less aggressively, since escalation at this rate indicates quality loss from quantisation."}],"correct":false,"tempting":[{"text":"Names one possible cause of weak answers without addressing the economics; even a perfect local pass is wasted if most work still goes out."}]},{"text":[{"text":"The escalation threshold is too strict and should be loosened until the deflection rate exceeds 80%."}],"correct":false,"tempting":[{"text":"Optimises the metric rather than the outcome, by pushing through local answers that the check has already judged inadequate."}]},{"text":[{"text":"The hosted model should be placed first with the local model as fallback, reversing the order."}],"correct":false,"tempting":[{"text":"Reverses the order without changing the arithmetic, and gives up the privacy and latency reasons for trying local first."}]},{"text":[{"text":"It costs more than calling the hosted model directly: every escalated request paid twice before paying the API."}],"correct":true,"tempting":null}],"order":[2,1,3,0],"why":null},"prev":{"slug":"sharing-one-gpu-fairly","title":"One card, several people","minutes":9},"next":{"slug":"the-honest-cost-comparison","title":"The honest cost comparison","minutes":10}}}