Chat Templates and Loss Masking

The template is a contract

An instruction-tuned model was trained with special tokens marking who is speaking and where a turn ends. Get them wrong by one character and you are training a slightly different model than the one you serve.

Never write the format by hand. Ask the tokenizer.

from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-7B-Instruct")
msgs = [{"role": "user", "content": "hi"},
        {"role": "assistant", "content": "hello"}]
print(repr(tok.apply_chat_template(msgs, tokenize=False)))

You will get something like '<|im_start|>user\nhi<|im_end|>\n<|im_start|>assistant\nhello<|im_end|>\n'. Every family differs: Llama uses <|start_header_id|> and <|eot_id|>, Gemma uses <start_of_turn>. Copying a snippet from a blog post about a different model is a common and silent way to lose a day.

Two rules:

• Training: render the full conversation including the assistant turn and its end token. add_generation_prompt=False.
• Inference: render up to the assistant header and stop. add_generation_prompt=True.

If you fine-tune a base model, with no -Instruct in the name, there is no template. You invent one, and you are then responsible for using exactly the same one at serving time. Write it down in the repo next to the weights.

Mask the prompt

By default many trainers compute loss on every token, the user’s message included. That teaches the model to generate user turns: wasted capacity at best, and a strong pull toward continuing past its own answer with an invented question.

Train on assistant tokens only. Every framework has a switch for this, and the names and APIs churn between versions, so do not trust the flag. Verify the labels:

batch = next(iter(trainer.get_train_dataloader()))
ids, labels = batch["input_ids"][0], batch["labels"][0]
print("TRAINED ON:", tok.decode([i for i, l in zip(ids, labels) if l != -100]))
print("MASKED   :", tok.decode([i for i, l in zip(ids, labels) if l == -100]))

Run this once, read both lines, and you have eliminated the majority of fine-tuning bugs. What should print under TRAINED ON is your assistant’s reply and its end-of-turn token, and nothing else.

The end token is the whole ballgame

The single most common report is: the model answers, then writes a new user message and answers that one too.

That is an end-token bug, essentially every time. One of:

• The end-of-turn token was stripped when you built the dataset.
• It fell inside the masked region, so no gradient ever taught the model to emit it.
• Your training template’s end token differs from the one your server stops on.
• max_length truncates long examples just before the end token, so long answers never see one.

Check it explicitly:

print(tok.decode(ids[-8:]))     # last tokens of a training example
print(labels[-1] != -100)       # is the final token actually trained on?

A stop string at the API layer hides the symptom. Fix the labels instead.

Small things that cost hours

• Double BOS. apply_chat_template usually adds the beginning-of-sequence token. If you then tokenize with add_special_tokens=True you get two, and the model sees an input shape it never saw in pretraining. Pass add_special_tokens=False after templating.
• Padding token. Many base models ship without one. Setting tok.pad_token = tok.eos_token is standard, but then make sure padding is masked in the labels, or you train the model to emit end-of-sequence forever.
• Packing. Concatenating short examples to fill the context is efficient, but naive packing lets one example attend to the next. Use it only if your trainer does position-aware packing with a correct attention mask, and skip it for short instruction data where the win is small anyway.
• System prompt. Whatever you train with, serve with. If your training examples carry no system prompt and production sends six hundred tokens of one, every request is off-distribution in a way that degrades everything.
• The data file. JSONL, one conversation per line, in the messages format. Let the trainer apply the template; do not pre-render strings unless you have to.

{"messages": [{"role": "user", "content": "..."}, {"role": "assistant", "content": "..."}]}

Do the label-decoding check before every run. It costs five seconds.