{"course":{"slug":"running-models-yourself","title":"Running Models Yourself","url":"https://addaly.com/learn/running-models-yourself","level":"intermediate","track":"deep"},"lesson":{"slug":"what-it-will-never-do","title":"The ceiling the weights impose","url":"https://addaly.com/learn/running-models-yourself/what-it-will-never-do","position":8,"of":82,"minutes":8,"answer":"Every capability limit of a local model lives in the weights, so design around it: retrieve rather than recall, one step per call, constrain and verify the output, and route the hard minority elsewhere.","keyIdea":"Every capability limit of a local model lives in the weights, so design around it: retrieve rather than recall, one step per call, constrain and verify the output, and route the hard minority elsewhere."},"body":{"format":"markdown","text":"## Hardware buys speed, not capability\n\nThis is the one sentence that saves the most wasted money in this subject. A faster machine runs the same weights more quickly. It does not make them better. Every capability limit below is a property of the model file, and no configuration, sampler, quantisation choice or graphics card changes it.\n\nWhat follows is the honest list for the 4B-to-32B band that most local deployments live in.\n\n## Long multi-step reasoning\n\nA small model can do two or three dependent steps. Past that, error compounds: each step is conditioned on the previous one, so a 90% per-step accuracy over six steps is about 53% overall. Reasoning variants improve this and do not remove it. The workaround is not a bigger machine, it is a shorter chain: decompose the task into steps you verify separately, and check each step in code rather than trusting the model to notice its own mistake.\n\n## Reliable tool calling in a loop\n\nSmall models emit tool calls with the right names and plausible arguments most of the time. \"Most of the time\" is the problem: an agent that takes ten actions with 92% per-call reliability completes about 43% of the time. Common failure shapes are calling a tool that does not exist, passing a string where a number is expected, calling the same tool repeatedly in a loop, and declaring success without having done anything.\n\nConstrained decoding fixes the syntax and not the judgement. If you are building agents locally, design for one or two tool calls per turn with validation in between, not for autonomous loops.\n\n## Recall over long context\n\nAdvertised context and useful context are different numbers. A model advertised at 128k typically retrieves a single fact placed anywhere in that window reasonably well (that is the easy version of the test), while degrading badly on anything requiring several facts from different places, or a fact that contradicts something earlier. Performance usually falls off long before the advertised limit, often somewhere between 8k and 32k for models in this band.\n\nThe practical consequence: retrieving five focused passages beats pasting a hundred pages, and it is cheaper.\n\n## Knowledge, and knowing that it does not know\n\nA small model has less compressed world knowledge, and its knowledge is much thinner outside English and outside prominent subjects. It will still answer confidently. Calibration (producing a lower probability when it is less sure) is weak at this size, so the model's confidence carries almost no information about whether it is right.\n\nThis is not fixable by prompting it to say when it is unsure. It will comply with the instruction stylistically and not epistemically. The fix is retrieval, so the answer comes from a document you can point at, or a verification step in code.\n\n## Exact arithmetic and counting\n\nDigits are tokens, and arithmetic over tokens is not arithmetic. Small models get multi-digit multiplication wrong, miscount items in a list, and mishandle dates. Give them a calculator, a Python interpreter, or a query, and check the result.\n\n## What it does very well\n\nThe list is not all limits. In the same band, local models are good at: classification into a known set of labels, extracting fields that are present in the text, rewriting for tone or length, summarising a document that fits in context, translating between high-resource languages, generating boilerplate code, tagging, and answering questions from retrieved passages.\n\nNotice what those have in common. The information needed is in the prompt, the output is short, and there is one step. That is the shape of task where a local model is not a compromise: it is the correct engineering choice, faster and cheaper than an API call and private by construction.\n\n## Designing around the ceiling\n\nThe systems that work locally share a pattern:\n\n1. **Retrieve, do not recall.** Put the facts in the prompt.\n2. **One step per call.** Chain in code, not inside the model.\n3. **Constrain the output.** A schema or a grammar, so the next stage can rely on the shape.\n4. **Verify in code.** Does the JSON parse, does the SQL run, does the citation exist, is the number in range.\n5. **Route the hard cases out.** A confidence signal or a rule sends the difficult 5% to a larger model or a human. This is the design that lets a small model carry a real workload honestly.\n\nA local system built this way is often more reliable than a frontier model used carelessly, because the verification is real. A local system that asks an 8B to plan, decide and act unsupervised will fail, and no amount of hardware is the reason."},"check":{"question":"An agent built on a local 8B chains eight tool calls per task and succeeds about 45% of the time. Per-call accuracy measured in isolation is around 92%. What does this tell you?","options":[{"text":"The failure is compounding across steps: 0.92 to the eighth power is roughly 0.44. So the fix is fewer steps with verification between them, not a better prompt or a faster card.","correct":true},{"text":"Tool-call accuracy degrades within a session as the context fills, so trimming conversation history should restore per-call reliability.","correct":false,"tempting":"Reaches for a context explanation when the arithmetic already accounts for the observed rate exactly, with no extra mechanism required."},{"text":"92% per-call accuracy is too low for tool use and constrained decoding with a JSON schema would raise it enough to fix the chain.","correct":false,"tempting":"Confuses syntactic validity with correct choices; grammars guarantee well-formed calls, not the right tool with the right arguments."},{"text":"Eight steps exceeds what the model can hold, so a longer context window and a larger KV cache would allow the chain to complete.","correct":false,"tempting":"Blames capacity for an error-rate problem; eight tool calls are a few thousand tokens, far inside the window."}]},"prev":{"slug":"the-hour-long-audition","title":"The audition: settling it for about a dollar","url":"https://addaly.com/learn/running-models-yourself/the-hour-long-audition"},"next":{"slug":"local-is-not-automatically-private","title":"Local is not automatically private","url":"https://addaly.com/learn/running-models-yourself/local-is-not-automatically-private"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'The ceiling the weights impose', in Running Models Yourself. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/running-models-yourself/what-it-will-never-do (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"The ceiling the weights impose","courseTitle":"Running Models Yourself","module":{"index":1,"slug":"deciding-to-run-it-yourself","title":"Deciding to run it yourself"},"answer":null,"keyIdea":"Every capability limit of a local model lives in the weights, so design around it: retrieve rather than recall, one step per call, constrain and verify the output, and route the hard minority elsewhere.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"hardware-buys-speed-not-capability","runs":[{"text":"Hardware buys speed, not capability"}]},{"t":"p","runs":[{"text":"This is the one sentence that saves the most wasted money in this subject. A faster machine runs the same weights more quickly. It does not make them better. Every capability limit below is a property of the model file, and no configuration, sampler, quantisation choice or graphics card changes it."}]},{"t":"p","runs":[{"text":"What follows is the honest list for the 4B-to-32B band that most local deployments live in."}]},{"t":"h2","id":"long-multi-step-reasoning","runs":[{"text":"Long multi-step reasoning"}]},{"t":"p","runs":[{"text":"A small model can do two or three dependent steps. Past that, error compounds: each step is conditioned on the previous one, so a 90% per-step accuracy over six steps is about 53% overall. Reasoning variants improve this and do not remove it. The workaround is not a bigger machine, it is a shorter chain: decompose the task into steps you verify separately, and check each step in code rather than trusting the model to notice its own mistake."}]},{"t":"h2","id":"reliable-tool-calling-in-a-loop","runs":[{"text":"Reliable tool calling in a loop"}]},{"t":"p","runs":[{"text":"Small models emit tool calls with the right names and plausible arguments most of the time. “Most of the time” is the problem: an agent that takes ten actions with 92% per-call reliability completes about 43% of the time. Common failure shapes are calling a tool that does not exist, passing a string where a number is expected, calling the same tool repeatedly in a loop, and declaring success without having done anything."}]},{"t":"p","runs":[{"text":"Constrained decoding fixes the syntax and not the judgement. If you are building agents locally, design for one or two tool calls per turn with validation in between, not for autonomous loops."}]},{"t":"h2","id":"recall-over-long-context","runs":[{"text":"Recall over long context"}]},{"t":"p","runs":[{"text":"Advertised context and useful context are different numbers. A model advertised at 128k typically retrieves a single fact placed anywhere in that window reasonably well (that is the easy version of the test), while degrading badly on anything requiring several facts from different places, or a fact that contradicts something earlier. Performance usually falls off long before the advertised limit, often somewhere between 8k and 32k for models in this band."}]},{"t":"p","runs":[{"text":"The practical consequence: retrieving five focused passages beats pasting a hundred pages, and it is cheaper."}]},{"t":"h2","id":"knowledge-and-knowing-that-it-does-not-know","runs":[{"text":"Knowledge, and knowing that it does not know"}]},{"t":"p","runs":[{"text":"A small model has less compressed world knowledge, and its knowledge is much thinner outside English and outside prominent subjects. It will still answer confidently. Calibration (producing a lower probability when it is less sure) is weak at this size, so the model’s confidence carries almost no information about whether it is right."}]},{"t":"p","runs":[{"text":"This is not fixable by prompting it to say when it is unsure. It will comply with the instruction stylistically and not epistemically. The fix is retrieval, so the answer comes from a document you can point at, or a verification step in code."}]},{"t":"h2","id":"exact-arithmetic-and-counting","runs":[{"text":"Exact arithmetic and counting"}]},{"t":"p","runs":[{"text":"Digits are tokens, and arithmetic over tokens is not arithmetic. Small models get multi-digit multiplication wrong, miscount items in a list, and mishandle dates. Give them a calculator, a Python interpreter, or a query, and check the result."}]},{"t":"h2","id":"what-it-does-very-well","runs":[{"text":"What it does very well"}]},{"t":"p","runs":[{"text":"The list is not all limits. In the same band, local models are good at: classification into a known set of labels, extracting fields that are present in the text, rewriting for tone or length, summarising a document that fits in context, translating between high-resource languages, generating boilerplate code, tagging, and answering questions from retrieved passages."}]},{"t":"p","runs":[{"text":"Notice what those have in common. The information needed is in the prompt, the output is short, and there is one step. That is the shape of task where a local model is not a compromise: it is the correct engineering choice, faster and cheaper than an API call and private by construction."}]},{"t":"h2","id":"designing-around-the-ceiling","runs":[{"text":"Designing around the ceiling"}]},{"t":"p","runs":[{"text":"The systems that work locally share a pattern:"}]},{"t":"ol","start":1,"items":[[{"text":"Retrieve, do not recall.","b":true},{"text":" Put the facts in the prompt."}],[{"text":"One step per call.","b":true},{"text":" Chain in code, not inside the model."}],[{"text":"Constrain the output.","b":true},{"text":" A schema or a grammar, so the next stage can rely on the shape."}],[{"text":"Verify in code.","b":true},{"text":" Does the JSON parse, does the SQL run, does the citation exist, is the number in range."}],[{"text":"Route the hard cases out.","b":true},{"text":" A confidence signal or a rule sends the difficult 5% to a larger model or a human. This is the design that lets a small model carry a real workload honestly."}]]},{"t":"p","runs":[{"text":"A local system built this way is often more reliable than a frontier model used carelessly, because the verification is real. A local system that asks an 8B to plan, decide and act unsupervised will fail, and no amount of hardware is the reason."}]}],"check":{"question":[{"text":"An agent built on a local 8B chains eight tool calls per task and succeeds about 45% of the time. Per-call accuracy measured in isolation is around 92%. What does this tell you?"}],"options":[{"text":[{"text":"The failure is compounding across steps: 0.92 to the eighth power is roughly 0.44. So the fix is fewer steps with verification between them, not a better prompt or a faster card."}],"correct":true,"tempting":null},{"text":[{"text":"Tool-call accuracy degrades within a session as the context fills, so trimming conversation history should restore per-call reliability."}],"correct":false,"tempting":[{"text":"Reaches for a context explanation when the arithmetic already accounts for the observed rate exactly, with no extra mechanism required."}]},{"text":[{"text":"92% per-call accuracy is too low for tool use and constrained decoding with a JSON schema would raise it enough to fix the chain."}],"correct":false,"tempting":[{"text":"Confuses syntactic validity with correct choices; grammars guarantee well-formed calls, not the right tool with the right arguments."}]},{"text":[{"text":"Eight steps exceeds what the model can hold, so a longer context window and a larger KV cache would allow the chain to complete."}],"correct":false,"tempting":[{"text":"Blames capacity for an error-rate problem; eight tool calls are a few thousand tokens, far inside the window."}]}],"order":[1,2,3,0],"why":null},"prev":{"slug":"the-hour-long-audition","title":"The audition: settling it for about a dollar","minutes":9},"next":{"slug":"local-is-not-automatically-private","title":"Local is not automatically private","minutes":8}}}