Multi-turn conversations and tool calls

Which turns carry loss

A ten-turn conversation contains five assistant messages. Three ways to train on it, and they are not equivalent:

Last turn only. Mask everything except the final assistant message. Simple, and it throws away four fifths of your supervision. For a 10-turn dialogue you get one training signal out of five available.

Every assistant turn. Mask all user and system and tool content; compute loss on every assistant message. Five signals from one conversation. This is what you almost always want.

Everything. Train on user turns too. Occasionally deliberate (if you want a model that can simulate a user, or for continued pretraining on dialogue) and usually a mistake, because you are teaching the model to write the human’s side.

Doing it correctly

Building the mask by hand means finding the character offsets of each assistant span after templating, which is fiddly and breaks whenever the template changes. Modern chat templates solve this by marking the spans themselves with {% generation %} blocks, and the tokenizer will then return an assistant mask:

out = tok.apply_chat_template(messages, return_dict=True,
                              return_assistant_tokens_mask=True)
labels = [t if m else -100
          for t, m in zip(out["input_ids"], out["assistant_masks"])]

If the model’s template has no generation markers, the collator has to locate the assistant response by string matching on the turn delimiters, which is what TRL’s completion-only collator does, and which is why it needs you to pass exactly the right response template string. Get that string wrong by one character and it matches nothing, every position is masked, and your loss is identically zero from step one. That symptom (loss exactly 0.0, not merely small) is nearly always this bug.

The exposure-bias problem in multi-turn

When training on turn five, the model conditions on turns one to four, which are the gold assistant messages from your dataset, not messages it produced. At inference it will be conditioning on its own turn four, which may be worse.

This mismatch is why multi-turn quality degrades over a long conversation even when each turn looks good in isolation. There is no clean fix in supervised fine-tuning; the mitigations are partial:

• Include some conversations in your training data where an earlier assistant turn was imperfect and the later turn recovers gracefully. This teaches recovery, which is a real skill and is absent from any dataset built only from ideal conversations.
• Evaluate multi-turn by actually running multi-turn: generate the model’s own turn three, then ask for turn four. Evaluating each turn against gold history overstates quality, sometimes substantially.

Tool calls

A tool-calling exchange has four kinds of message, and only one of them carries loss:

{"messages": [
  {"role": "user", "content": "What's the status of order 88412?"},
  {"role": "assistant", "content": null,
   "tool_calls": [{"id": "c1", "type": "function",
     "function": {"name": "get_order",
                  "arguments": "{\"order_id\": \"88412\"}"}}]},
  {"role": "tool", "tool_call_id": "c1",
   "content": "{\"status\": \"in_transit\", \"eta\": \"2026-09-24\"}"},
  {"role": "assistant", "content": "It's in transit, arriving on 24 September."}
]}

• The user message is input. Masked.
• The first assistant message is output: the decision to call a tool and the arguments. Not masked: this is the behaviour you are teaching.
• The tool message is input. Masked. It is the world’s response, not the model’s.
• The second assistant message is output. Not masked.

Training on the tool result is the commonest error here, and its effect is specific: the model learns to predict what the tool will say, and at inference it will sometimes produce a plausible fabricated tool result instead of calling anything. That is a hallucination you installed deliberately, by masking incorrectly.

Arguments are a string, and that matters

In the standard format, arguments is a JSON string, not a nested object. Serialisation details then become training signal: key order, whitespace, whether numbers are quoted. If half your examples have {"order_id": "88412"} and half have { "order_id" : "88412" }, the model learns both and will produce either.

Serialise every example with the same function and the same options:

json.dumps(args, separators=(", ", ": "), sort_keys=True)

What to include besides successes

A dataset of only successful tool calls produces a model that always calls a tool. Include:

• Questions answerable without any tool, where the assistant answers.
• Tool calls that return an error, with the assistant recovering or reporting it.
• Requests for a tool that does not exist, where the assistant says so.

Without these, the model has no example of not calling a tool, and it will reach for one when a user says hello.