The ceiling the weights impose

Hardware buys speed, not capability

This is the one sentence that saves the most wasted money in this subject. A faster machine runs the same weights more quickly. It does not make them better. Every capability limit below is a property of the model file, and no configuration, sampler, quantisation choice or graphics card changes it.

What follows is the honest list for the 4B-to-32B band that most local deployments live in.

Long multi-step reasoning

A small model can do two or three dependent steps. Past that, error compounds: each step is conditioned on the previous one, so a 90% per-step accuracy over six steps is about 53% overall. Reasoning variants improve this and do not remove it. The workaround is not a bigger machine, it is a shorter chain: decompose the task into steps you verify separately, and check each step in code rather than trusting the model to notice its own mistake.

Reliable tool calling in a loop

Small models emit tool calls with the right names and plausible arguments most of the time. “Most of the time” is the problem: an agent that takes ten actions with 92% per-call reliability completes about 43% of the time. Common failure shapes are calling a tool that does not exist, passing a string where a number is expected, calling the same tool repeatedly in a loop, and declaring success without having done anything.

Constrained decoding fixes the syntax and not the judgement. If you are building agents locally, design for one or two tool calls per turn with validation in between, not for autonomous loops.

Recall over long context

Advertised context and useful context are different numbers. A model advertised at 128k typically retrieves a single fact placed anywhere in that window reasonably well (that is the easy version of the test), while degrading badly on anything requiring several facts from different places, or a fact that contradicts something earlier. Performance usually falls off long before the advertised limit, often somewhere between 8k and 32k for models in this band.

The practical consequence: retrieving five focused passages beats pasting a hundred pages, and it is cheaper.

Knowledge, and knowing that it does not know

A small model has less compressed world knowledge, and its knowledge is much thinner outside English and outside prominent subjects. It will still answer confidently. Calibration (producing a lower probability when it is less sure) is weak at this size, so the model’s confidence carries almost no information about whether it is right.

This is not fixable by prompting it to say when it is unsure. It will comply with the instruction stylistically and not epistemically. The fix is retrieval, so the answer comes from a document you can point at, or a verification step in code.

Exact arithmetic and counting

Digits are tokens, and arithmetic over tokens is not arithmetic. Small models get multi-digit multiplication wrong, miscount items in a list, and mishandle dates. Give them a calculator, a Python interpreter, or a query, and check the result.

What it does very well

The list is not all limits. In the same band, local models are good at: classification into a known set of labels, extracting fields that are present in the text, rewriting for tone or length, summarising a document that fits in context, translating between high-resource languages, generating boilerplate code, tagging, and answering questions from retrieved passages.

Notice what those have in common. The information needed is in the prompt, the output is short, and there is one step. That is the shape of task where a local model is not a compromise: it is the correct engineering choice, faster and cheaper than an API call and private by construction.

Designing around the ceiling

The systems that work locally share a pattern:

1. Retrieve, do not recall. Put the facts in the prompt.
2. One step per call. Chain in code, not inside the model.
3. Constrain the output. A schema or a grammar, so the next stage can rely on the shape.
4. Verify in code. Does the JSON parse, does the SQL run, does the citation exist, is the number in range.
5. Route the hard cases out. A confidence signal or a rule sends the difficult 5% to a larger model or a human. This is the design that lets a small model carry a real workload honestly.

A local system built this way is often more reliable than a frontier model used carelessly, because the verification is real. A local system that asks an 8B to plan, decide and act unsupervised will fail, and no amount of hardware is the reason.