{"course":{"slug":"running-models-yourself","title":"Running Models Yourself","url":"https://addaly.com/learn/running-models-yourself","level":"intermediate","track":"deep"},"lesson":{"slug":"tool-calling-locally","title":"Tool calling that survives contact with a small model","url":"https://addaly.com/learn/running-models-yourself/tool-calling-locally","position":51,"of":82,"minutes":9,"answer":"Tool calling is ordinary generation plus a parser, so match the parser to the model family, constrain the syntax with a grammar, validate the meaning in code.","keyIdea":"Tool calling is ordinary generation plus a parser, so match the parser to the model family, constrain the syntax with a grammar, validate the meaning in code, and cap the chain: reliability compounds and no prompt fixes an exponent."},"body":{"format":"markdown","text":"## What a tool call actually is\n\nThere is no special mechanism. The model is shown a description of some functions, and it emits text in a particular format that the server parses into a structured call. Your code executes the function and puts the result back into the conversation as another message. The model then continues.\n\nEverything that goes wrong follows from that being ordinary text generation with a parser downstream.\n\n## The format problem\n\nEach model family was fine-tuned to emit tool calls in its own shape: some emit JSON inside special tokens, some emit Python-like syntax, some use a named block. The server must parse the shape the model produces:\n\n```bash\nllama-server -m model.gguf --jinja\n\nvllm serve <model> --enable-auto-tool-choice --tool-call-parser hermes\n```\n\nGetting the parser wrong produces a specific and confusing symptom: the model emits a perfectly good tool call, the server fails to recognise it, and the call is returned to the user as ordinary text content. If your model appears to be *describing* the function it would like to call rather than calling it, this is why.\n\nCheck the model card for the expected format and the engine's documentation for the matching parser name. This one setting accounts for most \"tool calling does not work with local models\" reports.\n\n> Figure: Per-call reliability compounds, and no prompt fixes an exponent. Line chart of Whole chain succeeds, per cent against Tool calls in the chain: Ninety-two per cent per call (1, 92) (2, 85) (4, 72) (6, 61) (8, 51) (10, 43); Ninety-seven per cent per call (1, 97) (2, 94) (4, 89) (6, 83) (8, 78) (10, 74). A good 8B emits a syntactically valid call with the right function name most of the time, and most of the time is exactly the problem. Constrained decoding removes the syntax failures and not the judgement ones, so the design that works is one or two calls per turn with validation in code between them, and a step budget.\n\n## Reliability, with the arithmetic\n\nA good 8B emits a syntactically valid call with the right function name most of the time. Most of the time is not enough for a loop. At 92% per call, a ten-step agent completes about 43% of the time; at 97%, about 74%. The exponent is unforgiving and no prompt fixes an exponent.\n\nThe common failure shapes, in rough order of frequency:\n\n1. Calling a function that does not exist, often a plausible-sounding invention.\n2. Passing the wrong type: a string where a number belongs, a date in the wrong format.\n3. Calling the same tool repeatedly with the same arguments.\n4. Announcing success without having called anything.\n5. Producing prose alongside the call that the parser then mangles.\n\n## Designing tools a small model can use\n\nMost of the reliability is in the interface, not the model.\n\n**Few tools.** Three to five. A model choosing among fifteen tools picks wrong far more often, and the definitions themselves consume a large share of the context.\n\n**Flat arguments.** Two or three parameters, primitive types. Nested objects and arrays of objects are where type errors cluster.\n\n**Enumerations instead of free strings.** `status: \"open\" | \"closed\" | \"pending\"` cannot be spelled wrong. Free-text arguments will be spelled every way.\n\n**Descriptive names.** `search_customer_by_email` is chosen correctly more often than `query`. The name is the strongest signal the model has.\n\n**One example per tool in its description.** The same demonstration principle as everywhere else.\n\n**Error messages that instruct.** When a call fails, return something the model can act on: \"date must be YYYY-MM-DD, received 3rd March\". A bare \"invalid argument\" produces another wrong guess.\n\n## Constrain the syntax, then validate the meaning\n\nGrammar-constrained decoding against your tool schema guarantees a parseable call with a real function name and correctly typed arguments. That removes failure shapes 1, 2 and 5 entirely, which is most of them.\n\nIt does not remove the judgement errors: the wrong tool, the right tool with wrong values, the unnecessary call. Validate in code before executing: does the identifier exist, is the amount within range, has this exact call already been made this turn. That last check kills the repetition loop, which is the failure most likely to run up a bill or send something twice.\n\n## Architecture that works at this size\n\n**One or two calls per turn, not an autonomous loop.** Return to code between steps and decide there.\n\n**A step budget.** Six steps maximum, then stop and report. Without one, a confused agent will loop until something times out.\n\n**Idempotent or read-only tools wherever possible.** If a tool can be called twice safely, the repetition failure becomes harmless instead of expensive.\n\n**Confirmation before anything irreversible.** Sending, paying, deleting, publishing. A local 8B should not be the last check before an irreversible action, and designing as though it might be is the mistake that produces incidents.\n\n**Log every call and every result.** When an agent misbehaves, the transcript is the only way to find out where the reasoning went wrong, and reconstructing it afterwards is impossible.\n\n## The summary\n\nLocal models can drive tools usefully within a narrow, well-typed, short-horizon design with validation between steps. They cannot yet be trusted with long autonomous loops, and the gap against frontier models is wider here than on almost any other task. Build for the first and you will get reliable systems; build for the second and you will spend your time debugging."},"check":{"question":"A local model appears to \"describe\" the function it wants to call in plain prose, and the client never receives a structured tool call. What is the most likely cause?","options":[{"text":"The model was not fine-tuned for tool use and is imitating tool-calling text it saw in training without emitting the actual format.","correct":false,"tempting":"Possible for a base model, but the described symptom, well-formed intent arriving as content, is the classic signature of a parsing failure rather than an untrained one."},{"text":"The tool definitions exceed the context window, so they are truncated and the model has nothing valid to call.","correct":false,"tempting":"Truncated definitions produce invented or malformed calls, not coherent prose descriptions of the intended call."},{"text":"The server's tool-call parser does not match the format this model family emits, so a valid call is passed through as ordinary content.","correct":true},{"text":"Temperature is too high, so the model is sampling away from the special tokens that begin a tool call.","correct":false,"tempting":"Would produce intermittent failure across runs rather than a consistent pattern of calls arriving as text."}]},"prev":{"slug":"few-shot-and-format-anchoring","title":"Examples, anchoring and getting the shape right","url":"https://addaly.com/learn/running-models-yourself/few-shot-and-format-anchoring"},"next":{"slug":"long-context-in-practice","title":"Long context in practice","url":"https://addaly.com/learn/running-models-yourself/long-context-in-practice"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Tool calling that survives contact with a small model', in Running Models Yourself. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/running-models-yourself/tool-calling-locally (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Tool calling that survives contact with a small model","courseTitle":"Running Models Yourself","module":{"index":6,"slug":"getting-good-output","title":"Getting good output"},"answer":null,"keyIdea":"Tool calling is ordinary generation plus a parser, so match the parser to the model family, constrain the syntax with a grammar, validate the meaning in code, and cap the chain: reliability compounds and no prompt fixes an exponent.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"type":"line","title":"Per-call reliability compounds, and no prompt fixes an exponent","series":[{"label":"Ninety-two per cent per call","points":[[1,92],[2,85],[4,72],[6,61],[8,51],[10,43]]},{"label":"Ninety-seven per cent per call","points":[[1,97],[2,94],[4,89],[6,83],[8,78],[10,74]]}],"xLabel":"Tool calls in the chain","yLabel":"Whole chain succeeds, per cent","caption":"A good 8B emits a syntactically valid call with the right function name most of the time, and most of the time is exactly the problem. Constrained decoding removes the syntax failures and not the judgement ones, so the design that works is one or two calls per turn with validation in code between them, and a step budget."},"alt":"Figure: Per-call reliability compounds, and no prompt fixes an exponent. Line chart of Whole chain succeeds, per cent against Tool calls in the chain: Ninety-two per cent per call (1, 92) (2, 85) (4, 72) (6, 61) (8, 51) (10, 43); Ninety-seven per cent per call (1, 97) (2, 94) (4, 89) (6, 83) (8, 78) (10, 74). A good 8B emits a syntactically valid call with the right function name most of the time, and most of the time is exactly the problem. Constrained decoding removes the syntax failures and not the judgement ones, so the design that works is one or two calls per turn with validation in code between them, and a step budget."}},"blocks":[{"t":"h2","id":"what-a-tool-call-actually-is","runs":[{"text":"What a tool call actually is"}]},{"t":"p","runs":[{"text":"There is no special mechanism. The model is shown a description of some functions, and it emits text in a particular format that the server parses into a structured call. Your code executes the function and puts the result back into the conversation as another message. The model then continues."}]},{"t":"p","runs":[{"text":"Everything that goes wrong follows from that being ordinary text generation with a parser downstream."}]},{"t":"h2","id":"the-format-problem","runs":[{"text":"The format problem"}]},{"t":"p","runs":[{"text":"Each model family was fine-tuned to emit tool calls in its own shape: some emit JSON inside special tokens, some emit Python-like syntax, some use a named block. The server must parse the shape the model produces:"}]},{"t":"code","lang":"bash","label":"Bash","text":"llama-server -m model.gguf --jinja\n\nvllm serve <model> --enable-auto-tool-choice --tool-call-parser hermes"},{"t":"p","runs":[{"text":"Getting the parser wrong produces a specific and confusing symptom: the model emits a perfectly good tool call, the server fails to recognise it, and the call is returned to the user as ordinary text content. If your model appears to be "},{"text":"describing","i":true},{"text":" the function it would like to call rather than calling it, this is why."}]},{"t":"p","runs":[{"text":"Check the model card for the expected format and the engine’s documentation for the matching parser name. This one setting accounts for most “tool calling does not work with local models” reports."}]},{"t":"h2","id":"reliability-with-the-arithmetic","runs":[{"text":"Reliability, with the arithmetic"}]},{"t":"p","runs":[{"text":"A good 8B emits a syntactically valid call with the right function name most of the time. Most of the time is not enough for a loop. At 92% per call, a ten-step agent completes about 43% of the time; at 97%, about 74%. The exponent is unforgiving and no prompt fixes an exponent."}]},{"t":"p","runs":[{"text":"The common failure shapes, in rough order of frequency:"}]},{"t":"ol","start":1,"items":[[{"text":"Calling a function that does not exist, often a plausible-sounding invention."}],[{"text":"Passing the wrong type: a string where a number belongs, a date in the wrong format."}],[{"text":"Calling the same tool repeatedly with the same arguments."}],[{"text":"Announcing success without having called anything."}],[{"text":"Producing prose alongside the call that the parser then mangles."}]]},{"t":"h2","id":"designing-tools-a-small-model-can-use","runs":[{"text":"Designing tools a small model can use"}]},{"t":"p","runs":[{"text":"Most of the reliability is in the interface, not the model."}]},{"t":"p","runs":[{"text":"Few tools.","b":true},{"text":" Three to five. A model choosing among fifteen tools picks wrong far more often, and the definitions themselves consume a large share of the context."}]},{"t":"p","runs":[{"text":"Flat arguments.","b":true},{"text":" Two or three parameters, primitive types. Nested objects and arrays of objects are where type errors cluster."}]},{"t":"p","runs":[{"text":"Enumerations instead of free strings.","b":true},{"text":" "},{"text":"status: \"open\" | \"closed\" | \"pending\"","code":true},{"text":" cannot be spelled wrong. Free-text arguments will be spelled every way."}]},{"t":"p","runs":[{"text":"Descriptive names.","b":true},{"text":" "},{"text":"search_customer_by_email","code":true},{"text":" is chosen correctly more often than "},{"text":"query","code":true},{"text":". The name is the strongest signal the model has."}]},{"t":"p","runs":[{"text":"One example per tool in its description.","b":true},{"text":" The same demonstration principle as everywhere else."}]},{"t":"p","runs":[{"text":"Error messages that instruct.","b":true},{"text":" When a call fails, return something the model can act on: “date must be YYYY-MM-DD, received 3rd March”. A bare “invalid argument” produces another wrong guess."}]},{"t":"h2","id":"constrain-the-syntax-then-validate-the-meaning","runs":[{"text":"Constrain the syntax, then validate the meaning"}]},{"t":"p","runs":[{"text":"Grammar-constrained decoding against your tool schema guarantees a parseable call with a real function name and correctly typed arguments. That removes failure shapes 1, 2 and 5 entirely, which is most of them."}]},{"t":"p","runs":[{"text":"It does not remove the judgement errors: the wrong tool, the right tool with wrong values, the unnecessary call. Validate in code before executing: does the identifier exist, is the amount within range, has this exact call already been made this turn. That last check kills the repetition loop, which is the failure most likely to run up a bill or send something twice."}]},{"t":"h2","id":"architecture-that-works-at-this-size","runs":[{"text":"Architecture that works at this size"}]},{"t":"p","runs":[{"text":"One or two calls per turn, not an autonomous loop.","b":true},{"text":" Return to code between steps and decide there."}]},{"t":"p","runs":[{"text":"A step budget.","b":true},{"text":" Six steps maximum, then stop and report. Without one, a confused agent will loop until something times out."}]},{"t":"p","runs":[{"text":"Idempotent or read-only tools wherever possible.","b":true},{"text":" If a tool can be called twice safely, the repetition failure becomes harmless instead of expensive."}]},{"t":"p","runs":[{"text":"Confirmation before anything irreversible.","b":true},{"text":" Sending, paying, deleting, publishing. A local 8B should not be the last check before an irreversible action, and designing as though it might be is the mistake that produces incidents."}]},{"t":"p","runs":[{"text":"Log every call and every result.","b":true},{"text":" When an agent misbehaves, the transcript is the only way to find out where the reasoning went wrong, and reconstructing it afterwards is impossible."}]},{"t":"h2","id":"the-summary","runs":[{"text":"The summary"}]},{"t":"p","runs":[{"text":"Local models can drive tools usefully within a narrow, well-typed, short-horizon design with validation between steps. They cannot yet be trusted with long autonomous loops, and the gap against frontier models is wider here than on almost any other task. Build for the first and you will get reliable systems; build for the second and you will spend your time debugging."}]}],"check":{"question":[{"text":"A local model appears to “describe” the function it wants to call in plain prose, and the client never receives a structured tool call. What is the most likely cause?"}],"options":[{"text":[{"text":"The model was not fine-tuned for tool use and is imitating tool-calling text it saw in training without emitting the actual format."}],"correct":false,"tempting":[{"text":"Possible for a base model, but the described symptom, well-formed intent arriving as content, is the classic signature of a parsing failure rather than an untrained one."}]},{"text":[{"text":"The tool definitions exceed the context window, so they are truncated and the model has nothing valid to call."}],"correct":false,"tempting":[{"text":"Truncated definitions produce invented or malformed calls, not coherent prose descriptions of the intended call."}]},{"text":[{"text":"The server’s tool-call parser does not match the format this model family emits, so a valid call is passed through as ordinary content."}],"correct":true,"tempting":null},{"text":[{"text":"Temperature is too high, so the model is sampling away from the special tokens that begin a tool call."}],"correct":false,"tempting":[{"text":"Would produce intermittent failure across runs rather than a consistent pattern of calls arriving as text."}]}],"order":[0,3,1,2],"why":null},"prev":{"slug":"few-shot-and-format-anchoring","title":"Examples, anchoring and getting the shape right","minutes":8},"next":{"slug":"long-context-in-practice","title":"Long context in practice","minutes":9}}}