Tool calling, and what breaks at 8B

The mechanism

Tool calling is not a special capability of the runtime. It is a format the model was trained to emit and the server was written to parse.

Your tool definitions go into the prompt, usually as JSON inside a system turn shaped by the chat template. The model, when it decides to call one, emits a specific pattern: a <tool_call> block for some families, a JSON object with particular keys for others, a Python-like call for others still. The serving layer recognises that pattern, extracts the call, hands it back to your code, and your code runs the function and returns the result as a new message with a tool role.

Two things must line up: the model must have been trained for tool use, and your server must implement that model’s parser. In vLLM you name it explicitly:

vllm serve Qwen/Qwen3-8B --enable-auto-tool-choice --tool-call-parser hermes

This is why the identical client code produces perfect tool calls on one model and raw text containing a JSON blob on another. Nothing is broken; the parser does not match the format.

What works at 8B

• One to three tools, with names that describe the action plainly.
• Flat arguments: strings, numbers, enums. No nested objects.
• A single call per turn, then your code returns the result and the model continues.
• Clear disjoint purposes. Two tools that could both plausibly apply produce coin-flip selection.

What fails, reliably

Many tools. Selection accuracy falls off sharply as the list grows; beyond about ten tools a small model is close to guessing between similar ones. The fix is a routing step: use a cheap classifier or a retrieval step to select five candidate tools, then present only those.

Nested arguments. A parameter that is an object with its own fields is where malformed calls appear. Flatten it, even at the cost of an ugly interface.

Parallel calls. Emitting several calls at once is supported by some templates and unreliable at small sizes. Ask for one at a time.

Deciding not to call. A model given a hammer uses it. Questions answerable from general knowledge get a search call anyway. This needs an explicit instruction and an example of a no-call turn.

Argument fidelity. Dates get reformatted, identifiers get normalised, a currency gets dropped. Always validate the arguments against your schema before executing.

The pattern that makes it robust

Validate, and return failures to the model as tool results rather than as exceptions:

try:
    args = ToolArgs.model_validate_json(call.arguments)   # pydantic
    result = run(args)
except ValidationError as e:
    result = {"error": "invalid arguments", "detail": e.errors()[:3]}

messages.append({"role":"tool","tool_call_id":call.id,"content":json.dumps(result)})

A model that receives a clear error usually fixes its call on the second attempt. A model that receives nothing loops. Cap the retries at one or two, then fail loudly.

Two further safeguards worth building in from the start. Make every tool idempotent or guarded, because the model will occasionally call the same one twice. And never give a small model a tool that performs an irreversible action without a confirmation step: deleting, sending, paying. The compounding-error arithmetic from earlier applies directly, and a 92% correct tool selection is a 8% chance of an unwanted irreversible act per call.

The alternative that always works

If a model has no tool-calling training, or your server has no parser for it, you do not need either. Ask for constrained JSON and dispatch it yourself:

{"action": "search_orders", "customer_id": "...", "since": "2026-01-01"}

With a schema-constrained decoder the output is guaranteed valid, you write the dispatch in ten lines, and it works identically on every model including base models. You lose the multi-turn conventions the trained format provides, and for a one-shot decision that is no loss at all.

Start there when you are evaluating models. It removes the parser as a variable, so you are comparing models rather than comparing serving-stack support.