Tool calling that survives contact with a small model

What a tool call actually is

There is no special mechanism. The model is shown a description of some functions, and it emits text in a particular format that the server parses into a structured call. Your code executes the function and puts the result back into the conversation as another message. The model then continues.

Everything that goes wrong follows from that being ordinary text generation with a parser downstream.

The format problem

Each model family was fine-tuned to emit tool calls in its own shape: some emit JSON inside special tokens, some emit Python-like syntax, some use a named block. The server must parse the shape the model produces:

llama-server -m model.gguf --jinja

vllm serve <model> --enable-auto-tool-choice --tool-call-parser hermes

Getting the parser wrong produces a specific and confusing symptom: the model emits a perfectly good tool call, the server fails to recognise it, and the call is returned to the user as ordinary text content. If your model appears to be describing the function it would like to call rather than calling it, this is why.

Check the model card for the expected format and the engine’s documentation for the matching parser name. This one setting accounts for most “tool calling does not work with local models” reports.

Reliability, with the arithmetic

A good 8B emits a syntactically valid call with the right function name most of the time. Most of the time is not enough for a loop. At 92% per call, a ten-step agent completes about 43% of the time; at 97%, about 74%. The exponent is unforgiving and no prompt fixes an exponent.

The common failure shapes, in rough order of frequency:

1. Calling a function that does not exist, often a plausible-sounding invention.
2. Passing the wrong type: a string where a number belongs, a date in the wrong format.
3. Calling the same tool repeatedly with the same arguments.
4. Announcing success without having called anything.
5. Producing prose alongside the call that the parser then mangles.

Designing tools a small model can use

Most of the reliability is in the interface, not the model.

Few tools. Three to five. A model choosing among fifteen tools picks wrong far more often, and the definitions themselves consume a large share of the context.

Flat arguments. Two or three parameters, primitive types. Nested objects and arrays of objects are where type errors cluster.

Enumerations instead of free strings. status: "open" | "closed" | "pending" cannot be spelled wrong. Free-text arguments will be spelled every way.

Descriptive names. search_customer_by_email is chosen correctly more often than query. The name is the strongest signal the model has.

One example per tool in its description. The same demonstration principle as everywhere else.

Error messages that instruct. When a call fails, return something the model can act on: “date must be YYYY-MM-DD, received 3rd March”. A bare “invalid argument” produces another wrong guess.

Constrain the syntax, then validate the meaning

Grammar-constrained decoding against your tool schema guarantees a parseable call with a real function name and correctly typed arguments. That removes failure shapes 1, 2 and 5 entirely, which is most of them.

It does not remove the judgement errors: the wrong tool, the right tool with wrong values, the unnecessary call. Validate in code before executing: does the identifier exist, is the amount within range, has this exact call already been made this turn. That last check kills the repetition loop, which is the failure most likely to run up a bill or send something twice.

Architecture that works at this size

One or two calls per turn, not an autonomous loop. Return to code between steps and decide there.

A step budget. Six steps maximum, then stop and report. Without one, a confused agent will loop until something times out.

Idempotent or read-only tools wherever possible. If a tool can be called twice safely, the repetition failure becomes harmless instead of expensive.

Confirmation before anything irreversible. Sending, paying, deleting, publishing. A local 8B should not be the last check before an irreversible action, and designing as though it might be is the mistake that produces incidents.

Log every call and every result. When an agent misbehaves, the transcript is the only way to find out where the reasoning went wrong, and reconstructing it afterwards is impossible.

The summary

Local models can drive tools usefully within a narrow, well-typed, short-horizon design with validation between steps. They cannot yet be trusted with long autonomous loops, and the gap against frontier models is wider here than on almost any other task. Build for the first and you will get reliable systems; build for the second and you will spend your time debugging.