{"course":{"slug":"fine-tuning","title":"Fine-Tuning, and When Not To","url":"https://addaly.com/learn/fine-tuning","level":"advanced","track":"deep"},"lesson":{"slug":"templates-and-masking","title":"Chat Templates and Loss Masking","url":"https://addaly.com/learn/fine-tuning/templates-and-masking","position":48,"of":84,"minutes":9,"answer":"Decode one batch's unmasked labels before every run; it catches most fine-tuning bugs in five seconds.","keyIdea":"Decode one batch's unmasked labels before every run; it catches most fine-tuning bugs in five seconds."},"body":{"format":"markdown","text":"## The template is a contract\n\nAn instruction-tuned model was trained with special tokens marking who is speaking and where a turn ends. Get them wrong by one character and you are training a slightly different model than the one you serve.\n\nNever write the format by hand. Ask the tokenizer.\n\n```python\nfrom transformers import AutoTokenizer\ntok = AutoTokenizer.from_pretrained(\"Qwen/Qwen2.5-7B-Instruct\")\nmsgs = [{\"role\": \"user\", \"content\": \"hi\"},\n        {\"role\": \"assistant\", \"content\": \"hello\"}]\nprint(repr(tok.apply_chat_template(msgs, tokenize=False)))\n```\n\nYou will get something like `'<|im_start|>user\\nhi<|im_end|>\\n<|im_start|>assistant\\nhello<|im_end|>\\n'`. Every family differs: Llama uses `<|start_header_id|>` and `<|eot_id|>`, Gemma uses `<start_of_turn>`. Copying a snippet from a blog post about a different model is a common and silent way to lose a day.\n\nTwo rules:\n\n- **Training**: render the full conversation including the assistant turn and its end token. `add_generation_prompt=False`.\n- **Inference**: render up to the assistant header and stop. `add_generation_prompt=True`.\n\nIf you fine-tune a **base** model, with no `-Instruct` in the name, there is no template. You invent one, and you are then responsible for using exactly the same one at serving time. Write it down in the repo next to the weights.\n\n## Mask the prompt\n\nBy default many trainers compute loss on every token, the user's message included. That teaches the model to generate user turns: wasted capacity at best, and a strong pull toward continuing past its own answer with an invented question.\n\nTrain on assistant tokens only. Every framework has a switch for this, and the names and APIs churn between versions, so do not trust the flag. Verify the labels:\n\n```python\nbatch = next(iter(trainer.get_train_dataloader()))\nids, labels = batch[\"input_ids\"][0], batch[\"labels\"][0]\nprint(\"TRAINED ON:\", tok.decode([i for i, l in zip(ids, labels) if l != -100]))\nprint(\"MASKED   :\", tok.decode([i for i, l in zip(ids, labels) if l == -100]))\n```\n\nRun this once, read both lines, and you have eliminated the majority of fine-tuning bugs. What should print under TRAINED ON is your assistant's reply and its end-of-turn token, and nothing else.\n\n## The end token is the whole ballgame\n\nThe single most common report is: the model answers, then writes a new user message and answers that one too.\n\nThat is an end-token bug, essentially every time. One of:\n\n- The end-of-turn token was stripped when you built the dataset.\n- It fell inside the masked region, so no gradient ever taught the model to emit it.\n- Your training template's end token differs from the one your server stops on.\n- `max_length` truncates long examples just before the end token, so long answers never see one.\n\nCheck it explicitly:\n\n```python\nprint(tok.decode(ids[-8:]))     # last tokens of a training example\nprint(labels[-1] != -100)       # is the final token actually trained on?\n```\n\nA stop string at the API layer hides the symptom. Fix the labels instead.\n\n## Small things that cost hours\n\n- **Double BOS.** `apply_chat_template` usually adds the beginning-of-sequence token. If you then tokenize with `add_special_tokens=True` you get two, and the model sees an input shape it never saw in pretraining. Pass `add_special_tokens=False` after templating.\n- **Padding token.** Many base models ship without one. Setting `tok.pad_token = tok.eos_token` is standard, but then make sure padding is masked in the labels, or you train the model to emit end-of-sequence forever.\n- **Packing.** Concatenating short examples to fill the context is efficient, but naive packing lets one example attend to the next. Use it only if your trainer does position-aware packing with a correct attention mask, and skip it for short instruction data where the win is small anyway.\n- **System prompt.** Whatever you train with, serve with. If your training examples carry no system prompt and production sends six hundred tokens of one, every request is off-distribution in a way that degrades everything.\n- **The data file.** JSONL, one conversation per line, in the `messages` format. Let the trainer apply the template; do not pre-render strings unless you have to.\n\n```json\n{\"messages\": [{\"role\": \"user\", \"content\": \"...\"}, {\"role\": \"assistant\", \"content\": \"...\"}]}\n```\n\nDo the label-decoding check before every run. It costs five seconds.\n\n> Figure: What the label check should print. TRAINED ON: the assistant's reply; its end-of-turn token; nothing else. MASKED: the system prompt; the user's message; every role header and delimiter; the padding, and check that padding is not the same id as end-of-sequence. Decode the unmasked labels of one real batch before every run. Five seconds, and it eliminates the majority of fine-tuning bugs."},"check":{"question":"Your fine-tune trained cleanly and the loss curve looks healthy. At inference the model answers the question, then writes a new user message and answers that one too. What is the most likely cause?","options":[{"text":"Sampling temperature is too high, so generation runs on past the answer.","correct":false,"tempting":"High temperature does produce rambling, but greedy decoding would run past the end in exactly the same way; the model has no learned reason to stop."},{"text":"The end-of-turn token was never trained on (stripped, masked out, or truncated away) so the model never learned where a turn stops.","correct":true},{"text":"The model needs more epochs to learn the conversation format.","correct":false,"tempting":"More epochs do sharpen formatting, but no number of epochs teaches a token that never appears in the loss."},{"text":"You need a stop sequence configured at the serving layer.","correct":false,"tempting":"A stop string does suppress the symptom and you should set one anyway, but it leaves a model that cannot end its own turn, and every downstream consumer inherits the workaround."}]},"prev":{"slug":"heads-instead-of-tokens","title":"Training a head instead of generating tokens","url":"https://addaly.com/learn/fine-tuning/heads-instead-of-tokens"},"next":{"slug":"the-run","title":"Running a LoRA, and What It Costs","url":"https://addaly.com/learn/fine-tuning/the-run"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Chat Templates and Loss Masking', in Fine-Tuning, and When Not To. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/fine-tuning/templates-and-masking (Accessed: 31 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Chat Templates and Loss Masking","courseTitle":"Fine-Tuning, and When Not To","module":{"index":6,"slug":"running-the-job","title":"The mechanics of a run"},"answer":null,"keyIdea":"Decode one batch’s unmasked labels before every run; it catches most fine-tuning bugs in five seconds.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure 6.1","figure":{"left":{"title":"TRAINED ON","items":["the assistant’s reply","its end-of-turn token","nothing else"]},"type":"compare","right":{"title":"MASKED","items":["the system prompt","the user’s message","every role header and delimiter","the padding, and check that padding is not the same id as end-of-sequence"]},"title":"What the label check should print","caption":"Decode the unmasked labels of one real batch before every run. Five seconds, and it eliminates the majority of fine-tuning bugs."},"alt":"Figure: What the label check should print. TRAINED ON: the assistant’s reply; its end-of-turn token; nothing else. MASKED: the system prompt; the user’s message; every role header and delimiter; the padding, and check that padding is not the same id as end-of-sequence. Decode the unmasked labels of one real batch before every run. Five seconds, and it eliminates the majority of fine-tuning bugs."}},"blocks":[{"t":"h2","id":"the-template-is-a-contract","runs":[{"text":"The template is a contract"}]},{"t":"p","runs":[{"text":"An instruction-tuned model was trained with special tokens marking who is speaking and where a turn ends. Get them wrong by one character and you are training a slightly different model than the one you serve."}]},{"t":"p","runs":[{"text":"Never write the format by hand. Ask the tokenizer."}]},{"t":"code","lang":"python","label":"Python","text":"from transformers import AutoTokenizer\ntok = AutoTokenizer.from_pretrained(\"Qwen/Qwen2.5-7B-Instruct\")\nmsgs = [{\"role\": \"user\", \"content\": \"hi\"},\n        {\"role\": \"assistant\", \"content\": \"hello\"}]\nprint(repr(tok.apply_chat_template(msgs, tokenize=False)))"},{"t":"p","runs":[{"text":"You will get something like "},{"text":"'<|im_start|>user\\nhi<|im_end|>\\n<|im_start|>assistant\\nhello<|im_end|>\\n'","code":true},{"text":". Every family differs: Llama uses "},{"text":"<|start_header_id|>","code":true},{"text":" and "},{"text":"<|eot_id|>","code":true},{"text":", Gemma uses "},{"text":"<start_of_turn>","code":true},{"text":". Copying a snippet from a blog post about a different model is a common and silent way to lose a day."}]},{"t":"p","runs":[{"text":"Two rules:"}]},{"t":"ul","items":[[{"text":"Training","b":true},{"text":": render the full conversation including the assistant turn and its end token. "},{"text":"add_generation_prompt=False","code":true},{"text":"."}],[{"text":"Inference","b":true},{"text":": render up to the assistant header and stop. "},{"text":"add_generation_prompt=True","code":true},{"text":"."}]]},{"t":"p","runs":[{"text":"If you fine-tune a "},{"text":"base","b":true},{"text":" model, with no "},{"text":"-Instruct","code":true},{"text":" in the name, there is no template. You invent one, and you are then responsible for using exactly the same one at serving time. Write it down in the repo next to the weights."}]},{"t":"h2","id":"mask-the-prompt","runs":[{"text":"Mask the prompt"}]},{"t":"p","runs":[{"text":"By default many trainers compute loss on every token, the user’s message included. That teaches the model to generate user turns: wasted capacity at best, and a strong pull toward continuing past its own answer with an invented question."}]},{"t":"p","runs":[{"text":"Train on assistant tokens only. Every framework has a switch for this, and the names and APIs churn between versions, so do not trust the flag. Verify the labels:"}]},{"t":"code","lang":"python","label":"Python","text":"batch = next(iter(trainer.get_train_dataloader()))\nids, labels = batch[\"input_ids\"][0], batch[\"labels\"][0]\nprint(\"TRAINED ON:\", tok.decode([i for i, l in zip(ids, labels) if l != -100]))\nprint(\"MASKED   :\", tok.decode([i for i, l in zip(ids, labels) if l == -100]))"},{"t":"p","runs":[{"text":"Run this once, read both lines, and you have eliminated the majority of fine-tuning bugs. What should print under TRAINED ON is your assistant’s reply and its end-of-turn token, and nothing else."}]},{"t":"h2","id":"the-end-token-is-the-whole-ballgame","runs":[{"text":"The end token is the whole ballgame"}]},{"t":"p","runs":[{"text":"The single most common report is: the model answers, then writes a new user message and answers that one too."}]},{"t":"p","runs":[{"text":"That is an end-token bug, essentially every time. One of:"}]},{"t":"ul","items":[[{"text":"The end-of-turn token was stripped when you built the dataset."}],[{"text":"It fell inside the masked region, so no gradient ever taught the model to emit it."}],[{"text":"Your training template’s end token differs from the one your server stops on."}],[{"text":"max_length","code":true},{"text":" truncates long examples just before the end token, so long answers never see one."}]]},{"t":"p","runs":[{"text":"Check it explicitly:"}]},{"t":"code","lang":"python","label":"Python","text":"print(tok.decode(ids[-8:]))     # last tokens of a training example\nprint(labels[-1] != -100)       # is the final token actually trained on?"},{"t":"p","runs":[{"text":"A stop string at the API layer hides the symptom. Fix the labels instead."}]},{"t":"h2","id":"small-things-that-cost-hours","runs":[{"text":"Small things that cost hours"}]},{"t":"ul","items":[[{"text":"Double BOS.","b":true},{"text":" "},{"text":"apply_chat_template","code":true},{"text":" usually adds the beginning-of-sequence token. If you then tokenize with "},{"text":"add_special_tokens=True","code":true},{"text":" you get two, and the model sees an input shape it never saw in pretraining. Pass "},{"text":"add_special_tokens=False","code":true},{"text":" after templating."}],[{"text":"Padding token.","b":true},{"text":" Many base models ship without one. Setting "},{"text":"tok.pad_token = tok.eos_token","code":true},{"text":" is standard, but then make sure padding is masked in the labels, or you train the model to emit end-of-sequence forever."}],[{"text":"Packing.","b":true},{"text":" Concatenating short examples to fill the context is efficient, but naive packing lets one example attend to the next. Use it only if your trainer does position-aware packing with a correct attention mask, and skip it for short instruction data where the win is small anyway."}],[{"text":"System prompt.","b":true},{"text":" Whatever you train with, serve with. If your training examples carry no system prompt and production sends six hundred tokens of one, every request is off-distribution in a way that degrades everything."}],[{"text":"The data file.","b":true},{"text":" JSONL, one conversation per line, in the "},{"text":"messages","code":true},{"text":" format. Let the trainer apply the template; do not pre-render strings unless you have to."}]]},{"t":"code","lang":"json","label":"JSON","text":"{\"messages\": [{\"role\": \"user\", \"content\": \"...\"}, {\"role\": \"assistant\", \"content\": \"...\"}]}"},{"t":"p","runs":[{"text":"Do the label-decoding check before every run. It costs five seconds."}]}],"check":{"question":[{"text":"Your fine-tune trained cleanly and the loss curve looks healthy. At inference the model answers the question, then writes a new user message and answers that one too. What is the most likely cause?"}],"options":[{"text":[{"text":"Sampling temperature is too high, so generation runs on past the answer."}],"correct":false,"tempting":[{"text":"High temperature does produce rambling, but greedy decoding would run past the end in exactly the same way; the model has no learned reason to stop."}]},{"text":[{"text":"The end-of-turn token was never trained on (stripped, masked out, or truncated away) so the model never learned where a turn stops."}],"correct":true,"tempting":null},{"text":[{"text":"The model needs more epochs to learn the conversation format."}],"correct":false,"tempting":[{"text":"More epochs do sharpen formatting, but no number of epochs teaches a token that never appears in the loss."}]},{"text":[{"text":"You need a stop sequence configured at the serving layer."}],"correct":false,"tempting":[{"text":"A stop string does suppress the symptom and you should set one anyway, but it leaves a model that cannot end its own turn, and every downstream consumer inherits the workaround."}]}],"order":[2,3,0,1],"why":null},"prev":{"slug":"heads-instead-of-tokens","title":"Training a head instead of generating tokens","minutes":9},"next":{"slug":"the-run","title":"Running a LoRA, and What It Costs","minutes":10}}}