{"course":{"slug":"fine-tuning","title":"Fine-Tuning, and When Not To","url":"https://addaly.com/learn/fine-tuning","level":"advanced","track":"deep"},"lesson":{"slug":"multi-turn-and-tool-calls","title":"Multi-turn conversations and tool calls","url":"https://addaly.com/learn/fine-tuning/multi-turn-and-tool-calls","position":36,"of":84,"minutes":9,"answer":"Compute loss on every assistant turn including the tool-call message, mask the tool's response because it is the world's output rather than the model's.","keyIdea":"Compute loss on every assistant turn including the tool-call message, mask the tool's response because it is the world's output rather than the model's, and include conversations where no tool was needed: a dataset of only successful calls produces a model that always calls something."},"body":{"format":"markdown","text":"## Which turns carry loss\n\nA ten-turn conversation contains five assistant messages. Three ways to train on it, and they are not equivalent:\n\n**Last turn only.** Mask everything except the final assistant message. Simple, and it throws away four fifths of your supervision. For a 10-turn dialogue you get one training signal out of five available.\n\n**Every assistant turn.** Mask all user and system and tool content; compute loss on every assistant message. Five signals from one conversation. This is what you almost always want.\n\n**Everything.** Train on user turns too. Occasionally deliberate (if you want a model that can simulate a user, or for continued pretraining on dialogue) and usually a mistake, because you are teaching the model to write the human's side.\n\n## Doing it correctly\n\nBuilding the mask by hand means finding the character offsets of each assistant span after templating, which is fiddly and breaks whenever the template changes. Modern chat templates solve this by marking the spans themselves with `{% generation %}` blocks, and the tokenizer will then return an assistant mask:\n\n```python\nout = tok.apply_chat_template(messages, return_dict=True,\n                              return_assistant_tokens_mask=True)\nlabels = [t if m else -100\n          for t, m in zip(out[\"input_ids\"], out[\"assistant_masks\"])]\n```\n\nIf the model's template has no generation markers, the collator has to locate the assistant response by string matching on the turn delimiters, which is what TRL's completion-only collator does, and which is why it needs you to pass exactly the right response template string. Get that string wrong by one character and it matches nothing, every position is masked, and your loss is identically zero from step one. That symptom (loss exactly 0.0, not merely small) is nearly always this bug.\n\n## The exposure-bias problem in multi-turn\n\nWhen training on turn five, the model conditions on turns one to four, which are the *gold* assistant messages from your dataset, not messages it produced. At inference it will be conditioning on its own turn four, which may be worse.\n\nThis mismatch is why multi-turn quality degrades over a long conversation even when each turn looks good in isolation. There is no clean fix in supervised fine-tuning; the mitigations are partial:\n\n- Include some conversations in your training data where an earlier assistant turn was imperfect and the later turn recovers gracefully. This teaches recovery, which is a real skill and is absent from any dataset built only from ideal conversations.\n- Evaluate multi-turn by actually running multi-turn: generate the model's own turn three, then ask for turn four. Evaluating each turn against gold history overstates quality, sometimes substantially.\n\n## Tool calls\n\nA tool-calling exchange has four kinds of message, and only one of them carries loss:\n\n```json\n{\"messages\": [\n  {\"role\": \"user\", \"content\": \"What's the status of order 88412?\"},\n  {\"role\": \"assistant\", \"content\": null,\n   \"tool_calls\": [{\"id\": \"c1\", \"type\": \"function\",\n     \"function\": {\"name\": \"get_order\",\n                  \"arguments\": \"{\\\"order_id\\\": \\\"88412\\\"}\"}}]},\n  {\"role\": \"tool\", \"tool_call_id\": \"c1\",\n   \"content\": \"{\\\"status\\\": \\\"in_transit\\\", \\\"eta\\\": \\\"2026-09-24\\\"}\"},\n  {\"role\": \"assistant\", \"content\": \"It's in transit, arriving on 24 September.\"}\n]}\n```\n\n- The **user** message is input. Masked.\n- The **first assistant** message is output: the decision to call a tool and the arguments. **Not masked**: this is the behaviour you are teaching.\n- The **tool** message is input. Masked. It is the world's response, not the model's.\n- The **second assistant** message is output. **Not masked.**\n\nTraining on the tool result is the commonest error here, and its effect is specific: the model learns to *predict* what the tool will say, and at inference it will sometimes produce a plausible fabricated tool result instead of calling anything. That is a hallucination you installed deliberately, by masking incorrectly.\n\n## Arguments are a string, and that matters\n\nIn the standard format, `arguments` is a JSON *string*, not a nested object. Serialisation details then become training signal: key order, whitespace, whether numbers are quoted. If half your examples have `{\"order_id\": \"88412\"}` and half have `{ \"order_id\" : \"88412\" }`, the model learns both and will produce either.\n\nSerialise every example with the same function and the same options:\n\n```python\njson.dumps(args, separators=(\", \", \": \"), sort_keys=True)\n```\n\n## What to include besides successes\n\nA dataset of only successful tool calls produces a model that always calls a tool. Include:\n\n- Questions answerable without any tool, where the assistant answers.\n- Tool calls that return an error, with the assistant recovering or reporting it.\n- Requests for a tool that does not exist, where the assistant says so.\n\nWithout these, the model has no example of *not* calling a tool, and it will reach for one when a user says hello."},"check":{"question":"A team includes the tool result messages in the training loss rather than masking them. What behaviour does this install?","options":[{"text":"The model refuses to call tools, having learned that tool results appear without a call preceding them.","correct":false,"tempting":"Reverses the effect; the call messages are still trained on, so the model retains the calling behaviour it was shown."},{"text":"The loss becomes dominated by JSON syntax tokens, so the model's natural-language turns degrade.","correct":false,"tempting":"Names a plausible-sounding shift in loss composition, but the distinctive failure is fabricated tool output, not degraded prose."},{"text":"Nothing, since tool results are deterministic and the model simply learns them as facts.","correct":false,"tempting":"Assumes the results are memorisable constants, when they are per-request data that the model can only imitate the shape of."},{"text":"The model learns to predict tool outputs and will sometimes fabricate one instead of calling.","correct":true}]},"prev":{"slug":"packing-and-long-context","title":"Packing, document boundaries and long context","url":"https://addaly.com/learn/fine-tuning/packing-and-long-context"},"next":{"slug":"privacy-and-memorisation","title":"Memorisation, and what you cannot take back","url":"https://addaly.com/learn/fine-tuning/privacy-and-memorisation"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Multi-turn conversations and tool calls', in Fine-Tuning, and When Not To. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/fine-tuning/multi-turn-and-tool-calls (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Multi-turn conversations and tool calls","courseTitle":"Fine-Tuning, and When Not To","module":{"index":4,"slug":"building-the-dataset","title":"Building the dataset"},"answer":null,"keyIdea":"Compute loss on every assistant turn including the tool-call message, mask the tool’s response because it is the world’s output rather than the model’s, and include conversations where no tool was needed: a dataset of only successful calls produces a model that always calls something.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"which-turns-carry-loss","runs":[{"text":"Which turns carry loss"}]},{"t":"p","runs":[{"text":"A ten-turn conversation contains five assistant messages. Three ways to train on it, and they are not equivalent:"}]},{"t":"p","runs":[{"text":"Last turn only.","b":true},{"text":" Mask everything except the final assistant message. Simple, and it throws away four fifths of your supervision. For a 10-turn dialogue you get one training signal out of five available."}]},{"t":"p","runs":[{"text":"Every assistant turn.","b":true},{"text":" Mask all user and system and tool content; compute loss on every assistant message. Five signals from one conversation. This is what you almost always want."}]},{"t":"p","runs":[{"text":"Everything.","b":true},{"text":" Train on user turns too. Occasionally deliberate (if you want a model that can simulate a user, or for continued pretraining on dialogue) and usually a mistake, because you are teaching the model to write the human’s side."}]},{"t":"h2","id":"doing-it-correctly","runs":[{"text":"Doing it correctly"}]},{"t":"p","runs":[{"text":"Building the mask by hand means finding the character offsets of each assistant span after templating, which is fiddly and breaks whenever the template changes. Modern chat templates solve this by marking the spans themselves with "},{"text":"{% generation %}","code":true},{"text":" blocks, and the tokenizer will then return an assistant mask:"}]},{"t":"code","lang":"python","label":"Python","text":"out = tok.apply_chat_template(messages, return_dict=True,\n                              return_assistant_tokens_mask=True)\nlabels = [t if m else -100\n          for t, m in zip(out[\"input_ids\"], out[\"assistant_masks\"])]"},{"t":"p","runs":[{"text":"If the model’s template has no generation markers, the collator has to locate the assistant response by string matching on the turn delimiters, which is what TRL’s completion-only collator does, and which is why it needs you to pass exactly the right response template string. Get that string wrong by one character and it matches nothing, every position is masked, and your loss is identically zero from step one. That symptom (loss exactly 0.0, not merely small) is nearly always this bug."}]},{"t":"h2","id":"the-exposure-bias-problem-in-multi-turn","runs":[{"text":"The exposure-bias problem in multi-turn"}]},{"t":"p","runs":[{"text":"When training on turn five, the model conditions on turns one to four, which are the "},{"text":"gold","i":true},{"text":" assistant messages from your dataset, not messages it produced. At inference it will be conditioning on its own turn four, which may be worse."}]},{"t":"p","runs":[{"text":"This mismatch is why multi-turn quality degrades over a long conversation even when each turn looks good in isolation. There is no clean fix in supervised fine-tuning; the mitigations are partial:"}]},{"t":"ul","items":[[{"text":"Include some conversations in your training data where an earlier assistant turn was imperfect and the later turn recovers gracefully. This teaches recovery, which is a real skill and is absent from any dataset built only from ideal conversations."}],[{"text":"Evaluate multi-turn by actually running multi-turn: generate the model’s own turn three, then ask for turn four. Evaluating each turn against gold history overstates quality, sometimes substantially."}]]},{"t":"h2","id":"tool-calls","runs":[{"text":"Tool calls"}]},{"t":"p","runs":[{"text":"A tool-calling exchange has four kinds of message, and only one of them carries loss:"}]},{"t":"code","lang":"json","label":"JSON","text":"{\"messages\": [\n  {\"role\": \"user\", \"content\": \"What's the status of order 88412?\"},\n  {\"role\": \"assistant\", \"content\": null,\n   \"tool_calls\": [{\"id\": \"c1\", \"type\": \"function\",\n     \"function\": {\"name\": \"get_order\",\n                  \"arguments\": \"{\\\"order_id\\\": \\\"88412\\\"}\"}}]},\n  {\"role\": \"tool\", \"tool_call_id\": \"c1\",\n   \"content\": \"{\\\"status\\\": \\\"in_transit\\\", \\\"eta\\\": \\\"2026-09-24\\\"}\"},\n  {\"role\": \"assistant\", \"content\": \"It's in transit, arriving on 24 September.\"}\n]}"},{"t":"ul","items":[[{"text":"The "},{"text":"user","b":true},{"text":" message is input. Masked."}],[{"text":"The "},{"text":"first assistant","b":true},{"text":" message is output: the decision to call a tool and the arguments. "},{"text":"Not masked","b":true},{"text":": this is the behaviour you are teaching."}],[{"text":"The "},{"text":"tool","b":true},{"text":" message is input. Masked. It is the world’s response, not the model’s."}],[{"text":"The "},{"text":"second assistant","b":true},{"text":" message is output. "},{"text":"Not masked.","b":true}]]},{"t":"p","runs":[{"text":"Training on the tool result is the commonest error here, and its effect is specific: the model learns to "},{"text":"predict","i":true},{"text":" what the tool will say, and at inference it will sometimes produce a plausible fabricated tool result instead of calling anything. That is a hallucination you installed deliberately, by masking incorrectly."}]},{"t":"h2","id":"arguments-are-a-string-and-that-matters","runs":[{"text":"Arguments are a string, and that matters"}]},{"t":"p","runs":[{"text":"In the standard format, "},{"text":"arguments","code":true},{"text":" is a JSON "},{"text":"string","i":true},{"text":", not a nested object. Serialisation details then become training signal: key order, whitespace, whether numbers are quoted. If half your examples have "},{"text":"{\"order_id\": \"88412\"}","code":true},{"text":" and half have "},{"text":"{ \"order_id\" : \"88412\" }","code":true},{"text":", the model learns both and will produce either."}]},{"t":"p","runs":[{"text":"Serialise every example with the same function and the same options:"}]},{"t":"code","lang":"python","label":"Python","text":"json.dumps(args, separators=(\", \", \": \"), sort_keys=True)"},{"t":"h2","id":"what-to-include-besides-successes","runs":[{"text":"What to include besides successes"}]},{"t":"p","runs":[{"text":"A dataset of only successful tool calls produces a model that always calls a tool. Include:"}]},{"t":"ul","items":[[{"text":"Questions answerable without any tool, where the assistant answers."}],[{"text":"Tool calls that return an error, with the assistant recovering or reporting it."}],[{"text":"Requests for a tool that does not exist, where the assistant says so."}]]},{"t":"p","runs":[{"text":"Without these, the model has no example of "},{"text":"not","i":true},{"text":" calling a tool, and it will reach for one when a user says hello."}]}],"check":{"question":[{"text":"A team includes the tool result messages in the training loss rather than masking them. What behaviour does this install?"}],"options":[{"text":[{"text":"The model refuses to call tools, having learned that tool results appear without a call preceding them."}],"correct":false,"tempting":[{"text":"Reverses the effect; the call messages are still trained on, so the model retains the calling behaviour it was shown."}]},{"text":[{"text":"The loss becomes dominated by JSON syntax tokens, so the model’s natural-language turns degrade."}],"correct":false,"tempting":[{"text":"Names a plausible-sounding shift in loss composition, but the distinctive failure is fabricated tool output, not degraded prose."}]},{"text":[{"text":"Nothing, since tool results are deterministic and the model simply learns them as facts."}],"correct":false,"tempting":[{"text":"Assumes the results are memorisable constants, when they are per-request data that the model can only imitate the shape of."}]},{"text":[{"text":"The model learns to predict tool outputs and will sometimes fabricate one instead of calling."}],"correct":true,"tempting":null}],"order":[2,1,0,3],"why":null},"prev":{"slug":"packing-and-long-context","title":"Packing, document boundaries and long context","minutes":9},"next":{"slug":"privacy-and-memorisation","title":"Memorisation, and what you cannot take back","minutes":10}}}