{"course":{"slug":"open-models","title":"The Open Model Ecosystem","url":"https://addaly.com/learn/open-models","level":"intermediate","track":"deep"},"lesson":{"slug":"tool-calling-on-open-weights","title":"Tool calling, and what breaks at 8B","url":"https://addaly.com/learn/open-models/tool-calling-on-open-weights","position":45,"of":71,"minutes":9,"answer":"Tool calling is a trained output format plus a matching server-side parser, and small models fail specifically on long tool lists, nested arguments and deciding not to call.","keyIdea":"Tool calling is a trained output format plus a matching server-side parser, and small models fail specifically on long tool lists, nested arguments and deciding not to call, so validate arguments and return errors as tool results rather than exceptions."},"body":{"format":"markdown","text":"## The mechanism\n\nTool calling is not a special capability of the runtime. It is a format the model was trained to emit and the server was written to parse.\n\nYour tool definitions go into the prompt, usually as JSON inside a system turn shaped by the chat template. The model, when it decides to call one, emits a specific pattern: a `<tool_call>` block for some families, a JSON object with particular keys for others, a Python-like call for others still. The serving layer recognises that pattern, extracts the call, hands it back to your code, and your code runs the function and returns the result as a new message with a `tool` role.\n\nTwo things must line up: the model must have been trained for tool use, and your server must implement that model's parser. In vLLM you name it explicitly:\n\n```bash\nvllm serve Qwen/Qwen3-8B --enable-auto-tool-choice --tool-call-parser hermes\n```\n\nThis is why the identical client code produces perfect tool calls on one model and raw text containing a JSON blob on another. Nothing is broken; the parser does not match the format.\n\n## What works at 8B\n\n- **One to three tools**, with names that describe the action plainly.\n- **Flat arguments**: strings, numbers, enums. No nested objects.\n- **A single call per turn**, then your code returns the result and the model continues.\n- **Clear disjoint purposes.** Two tools that could both plausibly apply produce coin-flip selection.\n\n## What fails, reliably\n\n**Many tools.** Selection accuracy falls off sharply as the list grows; beyond about ten tools a small model is close to guessing between similar ones. The fix is a routing step: use a cheap classifier or a retrieval step to select five candidate tools, then present only those.\n\n**Nested arguments.** A parameter that is an object with its own fields is where malformed calls appear. Flatten it, even at the cost of an ugly interface.\n\n**Parallel calls.** Emitting several calls at once is supported by some templates and unreliable at small sizes. Ask for one at a time.\n\n**Deciding not to call.** A model given a hammer uses it. Questions answerable from general knowledge get a search call anyway. This needs an explicit instruction and an example of a no-call turn.\n\n**Argument fidelity.** Dates get reformatted, identifiers get normalised, a currency gets dropped. Always validate the arguments against your schema before executing.\n\n## The pattern that makes it robust\n\nValidate, and return failures to the model as tool results rather than as exceptions:\n\n```python\ntry:\n    args = ToolArgs.model_validate_json(call.arguments)   # pydantic\n    result = run(args)\nexcept ValidationError as e:\n    result = {\"error\": \"invalid arguments\", \"detail\": e.errors()[:3]}\n\nmessages.append({\"role\":\"tool\",\"tool_call_id\":call.id,\"content\":json.dumps(result)})\n```\n\nA model that receives a clear error usually fixes its call on the second attempt. A model that receives nothing loops. Cap the retries at one or two, then fail loudly.\n\nTwo further safeguards worth building in from the start. **Make every tool idempotent or guarded**, because the model will occasionally call the same one twice. And **never give a small model a tool that performs an irreversible action without a confirmation step**: deleting, sending, paying. The compounding-error arithmetic from earlier applies directly, and a 92% correct tool selection is a 8% chance of an unwanted irreversible act per call.\n\n## The alternative that always works\n\nIf a model has no tool-calling training, or your server has no parser for it, you do not need either. Ask for constrained JSON and dispatch it yourself:\n\n```json\n{\"action\": \"search_orders\", \"customer_id\": \"...\", \"since\": \"2026-01-01\"}\n```\n\nWith a schema-constrained decoder the output is guaranteed valid, you write the dispatch in ten lines, and it works identically on every model including base models. You lose the multi-turn conventions the trained format provides, and for a one-shot decision that is no loss at all.\n\n> Start there when you are evaluating models. It removes the parser as a variable, so you are comparing models rather than comparing serving-stack support."},"check":{"question":"A model emits a well-formed JSON tool call as ordinary assistant text, and the client never sees a structured call. What is the fault?","options":[{"text":"The model has not been trained for tool use and is imitating the format from the prompt.","correct":false,"tempting":"Possible, but an untrained model usually produces inconsistent or prose-wrapped output rather than a well-formed call in the expected shape."},{"text":"The tool definitions were not included in the chat template, so the model invented the format.","correct":false,"tempting":"If the model emitted the correct family-specific format, the definitions almost certainly did reach it."},{"text":"The serving layer has no parser configured for this model's tool-call format, so the correct output is passed through as content.","correct":true},{"text":"Constrained decoding is disabled, so the call is not being validated.","correct":false,"tempting":"Constrained decoding governs output validity; the output here is already valid and the problem is that nothing extracted it."}]},"prev":{"slug":"examples-in-the-prompt","title":"Examples, and how to choose them","url":"https://addaly.com/learn/open-models/examples-in-the-prompt"},"next":{"slug":"agents-on-small-models","title":"Agent loops on models that are not clever","url":"https://addaly.com/learn/open-models/agents-on-small-models"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Tool calling, and what breaks at 8B', in The Open Model Ecosystem. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/open-models/tool-calling-on-open-weights (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Tool calling, and what breaks at 8B","courseTitle":"The Open Model Ecosystem","module":{"index":5,"slug":"getting-good-output","title":"Getting good output"},"answer":null,"keyIdea":"Tool calling is a trained output format plus a matching server-side parser, and small models fail specifically on long tool lists, nested arguments and deciding not to call, so validate arguments and return errors as tool results rather than exceptions.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"the-mechanism","runs":[{"text":"The mechanism"}]},{"t":"p","runs":[{"text":"Tool calling is not a special capability of the runtime. It is a format the model was trained to emit and the server was written to parse."}]},{"t":"p","runs":[{"text":"Your tool definitions go into the prompt, usually as JSON inside a system turn shaped by the chat template. The model, when it decides to call one, emits a specific pattern: a "},{"text":"<tool_call>","code":true},{"text":" block for some families, a JSON object with particular keys for others, a Python-like call for others still. The serving layer recognises that pattern, extracts the call, hands it back to your code, and your code runs the function and returns the result as a new message with a "},{"text":"tool","code":true},{"text":" role."}]},{"t":"p","runs":[{"text":"Two things must line up: the model must have been trained for tool use, and your server must implement that model’s parser. In vLLM you name it explicitly:"}]},{"t":"code","lang":"bash","label":"Bash","text":"vllm serve Qwen/Qwen3-8B --enable-auto-tool-choice --tool-call-parser hermes"},{"t":"p","runs":[{"text":"This is why the identical client code produces perfect tool calls on one model and raw text containing a JSON blob on another. Nothing is broken; the parser does not match the format."}]},{"t":"h2","id":"what-works-at-8b","runs":[{"text":"What works at 8B"}]},{"t":"ul","items":[[{"text":"One to three tools","b":true},{"text":", with names that describe the action plainly."}],[{"text":"Flat arguments","b":true},{"text":": strings, numbers, enums. No nested objects."}],[{"text":"A single call per turn","b":true},{"text":", then your code returns the result and the model continues."}],[{"text":"Clear disjoint purposes.","b":true},{"text":" Two tools that could both plausibly apply produce coin-flip selection."}]]},{"t":"h2","id":"what-fails-reliably","runs":[{"text":"What fails, reliably"}]},{"t":"p","runs":[{"text":"Many tools.","b":true},{"text":" Selection accuracy falls off sharply as the list grows; beyond about ten tools a small model is close to guessing between similar ones. The fix is a routing step: use a cheap classifier or a retrieval step to select five candidate tools, then present only those."}]},{"t":"p","runs":[{"text":"Nested arguments.","b":true},{"text":" A parameter that is an object with its own fields is where malformed calls appear. Flatten it, even at the cost of an ugly interface."}]},{"t":"p","runs":[{"text":"Parallel calls.","b":true},{"text":" Emitting several calls at once is supported by some templates and unreliable at small sizes. Ask for one at a time."}]},{"t":"p","runs":[{"text":"Deciding not to call.","b":true},{"text":" A model given a hammer uses it. Questions answerable from general knowledge get a search call anyway. This needs an explicit instruction and an example of a no-call turn."}]},{"t":"p","runs":[{"text":"Argument fidelity.","b":true},{"text":" Dates get reformatted, identifiers get normalised, a currency gets dropped. Always validate the arguments against your schema before executing."}]},{"t":"h2","id":"the-pattern-that-makes-it-robust","runs":[{"text":"The pattern that makes it robust"}]},{"t":"p","runs":[{"text":"Validate, and return failures to the model as tool results rather than as exceptions:"}]},{"t":"code","lang":"python","label":"Python","text":"try:\n    args = ToolArgs.model_validate_json(call.arguments)   # pydantic\n    result = run(args)\nexcept ValidationError as e:\n    result = {\"error\": \"invalid arguments\", \"detail\": e.errors()[:3]}\n\nmessages.append({\"role\":\"tool\",\"tool_call_id\":call.id,\"content\":json.dumps(result)})"},{"t":"p","runs":[{"text":"A model that receives a clear error usually fixes its call on the second attempt. A model that receives nothing loops. Cap the retries at one or two, then fail loudly."}]},{"t":"p","runs":[{"text":"Two further safeguards worth building in from the start. "},{"text":"Make every tool idempotent or guarded","b":true},{"text":", because the model will occasionally call the same one twice. And "},{"text":"never give a small model a tool that performs an irreversible action without a confirmation step","b":true},{"text":": deleting, sending, paying. The compounding-error arithmetic from earlier applies directly, and a 92% correct tool selection is a 8% chance of an unwanted irreversible act per call."}]},{"t":"h2","id":"the-alternative-that-always-works","runs":[{"text":"The alternative that always works"}]},{"t":"p","runs":[{"text":"If a model has no tool-calling training, or your server has no parser for it, you do not need either. Ask for constrained JSON and dispatch it yourself:"}]},{"t":"code","lang":"json","label":"JSON","text":"{\"action\": \"search_orders\", \"customer_id\": \"...\", \"since\": \"2026-01-01\"}"},{"t":"p","runs":[{"text":"With a schema-constrained decoder the output is guaranteed valid, you write the dispatch in ten lines, and it works identically on every model including base models. You lose the multi-turn conventions the trained format provides, and for a one-shot decision that is no loss at all."}]},{"t":"quote","runs":[{"text":"Start there when you are evaluating models. It removes the parser as a variable, so you are comparing models rather than comparing serving-stack support."}]}],"check":{"question":[{"text":"A model emits a well-formed JSON tool call as ordinary assistant text, and the client never sees a structured call. What is the fault?"}],"options":[{"text":[{"text":"The model has not been trained for tool use and is imitating the format from the prompt."}],"correct":false,"tempting":[{"text":"Possible, but an untrained model usually produces inconsistent or prose-wrapped output rather than a well-formed call in the expected shape."}]},{"text":[{"text":"The tool definitions were not included in the chat template, so the model invented the format."}],"correct":false,"tempting":[{"text":"If the model emitted the correct family-specific format, the definitions almost certainly did reach it."}]},{"text":[{"text":"The serving layer has no parser configured for this model’s tool-call format, so the correct output is passed through as content."}],"correct":true,"tempting":null},{"text":[{"text":"Constrained decoding is disabled, so the call is not being validated."}],"correct":false,"tempting":[{"text":"Constrained decoding governs output validity; the output here is already valid and the problem is that nothing extracted it."}]}],"order":[0,3,1,2],"why":null},"prev":{"slug":"examples-in-the-prompt","title":"Examples, and how to choose them","minutes":8},"next":{"slug":"agents-on-small-models","title":"Agent loops on models that are not clever","minutes":9}}}