{"course":{"slug":"running-models-yourself","title":"Running Models Yourself","url":"https://addaly.com/learn/running-models-yourself","level":"intermediate","track":"deep"},"lesson":{"slug":"chat-templates","title":"Chat templates: the most common silent failure","url":"https://addaly.com/learn/running-models-yourself/chat-templates","position":32,"of":82,"minutes":10,"answer":"The model receives one formatted string, not a message list, and a wrong template or a missing stop token produces exactly the symptoms people blame on the weights: print the actual prompt before concluding anything.","keyIdea":"The model receives one formatted string, not a message list, and a wrong template or a missing stop token produces exactly the symptoms people blame on the weights: print the actual prompt before concluding anything."},"body":{"format":"markdown","text":"## The model never sees your messages\n\nA chat model does not receive a list of messages. It receives one string of tokens. Something has to convert `[{\"role\": \"user\", \"content\": \"hello\"}]` into that string, and the conversion has to match the format the model was fine-tuned on, exactly.\n\nThat conversion is the chat template. When it is wrong, the model is being addressed in a dialect it does not recognise. It still answers, models are robust, but noticeably worse: ignoring the system prompt, failing to stop, drifting out of character, or producing rambling text. This accounts for a large share of \"this model is stupid\" reports, and it is invisible unless you look.\n\n## What the string looks like\n\nA typical modern format:\n\n```\n<|im_start|>system\nYou are a careful assistant.<|im_end|>\n<|im_start|>user\nWhat is the boiling point of water?<|im_end|>\n<|im_start|>assistant\n```\n\nThe markers are single tokens in the model's vocabulary, learned during fine-tuning. The trailing `<|im_start|>assistant` is the cue to begin answering. Different families use different markers, and the differences are not cosmetic: a model trained on one set has no idea what another's means.\n\n## Applying it correctly\n\nThe template is stored with the model, so let the tokeniser apply it:\n\n```python\nfrom transformers import AutoTokenizer\ntok = AutoTokenizer.from_pretrained(\"Qwen/Qwen3-8B\")\n\nmsgs = [{\"role\": \"system\", \"content\": \"You are terse.\"},\n        {\"role\": \"user\", \"content\": \"Why is the sky blue?\"}]\n\n> Figure: Four ways a template fails without raising an error. Layers, top to bottom: A borrowed template (A community quant packaged with another family's markers); A doubled beginning-of-sequence token (The template emits one and the tokeniser adds another); A system role the model never had (Your system prompt is folded into the user turn, or dropped); The end marker missing from the stop list (It answers, then invents the next user turn and answers that). The model receives one formatted string, never a list of messages. Print the string before concluding anything about the weights: every failure here is visible in it, none of them raises an error, and together they account for a large share of reports that a model is stupid.\n\nprint(tok.apply_chat_template(msgs, tokenize=False,\n                              add_generation_prompt=True))\n```\n\nPrint it. Reading the actual string once teaches you more than any explanation, and it is the fastest diagnostic when something is off.\n\n`add_generation_prompt=True` appends the assistant cue. Omit it and the model has not been told it is its turn, and will often continue the user's message instead: a bug that looks like the model misunderstanding the question.\n\n## The three failures worth recognising\n\n**Double BOS.** Many templates already emit a beginning-of-sequence token. If the tokeniser then adds another, the model starts with a sequence it has never seen. Symptoms are subtle: slightly worse quality, occasionally much worse. When calling a tokeniser directly, `add_special_tokens=False` on already-templated text is the fix.\n\n**A borrowed template.** A community quant packaged with a template copied from a different model family. The markers are wrong, and the model is being spoken to in a foreign dialect. Dump the metadata and compare against the original repository's `tokenizer_config.json`.\n\n**A system role the model does not have.** Some models were trained without a system role. Passing one gets it silently folded into the first user message, or dropped. If your system prompt seems to have no effect, check whether the template actually places it anywhere.\n\n## Seeing what your server sends\n\nIf you use an OpenAI-compatible endpoint, the server applies the template for you and you never see the result. Two ways to look:\n\n```bash\n# llama-server logs the formatted prompt at higher verbosity\nllama-server -m model.gguf --verbose\n\n# or bypass chat entirely and send the raw string yourself\ncurl http://localhost:8080/v1/completions -H 'Content-Type: application/json' \\\n  -d '{\"prompt\":\"<|im_start|>user\\nhello<|im_end|>\\n<|im_start|>assistant\\n\",\n       \"max_tokens\":64}'\n```\n\nThe second is the diagnostic that settles it. If the raw call behaves correctly and the chat call does not, the template is your problem.\n\n## Overriding a template\n\nWhen the packaged template is wrong, replace it rather than working around it:\n\n```bash\nllama-server -m model.gguf --chat-template chatml\n# or supply a Jinja file\nllama-server -m model.gguf --chat-template-file ./correct.jinja\n```\n\nIn Ollama the template lives in the Modelfile, and `ollama show <model> --modelfile` prints it. Copy it, correct it, and rebuild the model with `ollama create`.\n\n## Stop tokens\n\nThe other half of the same problem. The template's end marker must be in the server's stop list, or generation runs past the end of the answer and the model begins writing the next turn of the conversation itself: inventing a user question and answering it. If your model will not shut up, this is why, and the fix is a stop sequence rather than a shorter `max_tokens`.\n\nGood engines read the stop tokens from the model's metadata. When a model is new, or a quant was packaged carelessly, they are missing and you must supply them:\n\n```json\n{\"stop\": [\"<|im_end|>\", \"<|endoftext|>\"]}\n```\n\nCheck this on the first run of any new model. It takes one prompt and it saves a long, confused investigation."},"check":{"question":"A model answers a question correctly, then writes a plausible follow-up question from the user and answers that too, continuing until it hits the token limit. What is the mechanism?","options":[{"text":"The repetition penalty is too low, so the model is not discouraged from re-entering the conversational pattern it just produced.","correct":false,"tempting":"Applies a sampler remedy to a termination problem; penalties change token probabilities, not whether generation stops at a boundary."},{"text":"The context window is larger than the model's trained length, so it continues generating to fill the space it was given.","correct":false,"tempting":"Treats the window as a target to fill, which it is not; models stop at their end token regardless of remaining space."},{"text":"The end-of-turn marker is not registered as a stop sequence, so generation continues past it and the model simply predicts the next thing that would follow: another turn.","correct":true},{"text":"The chat template omitted the generation prompt, so the model is completing the user's message rather than replying.","correct":false,"tempting":"Names a real template bug with a different signature: that failure produces a continuation instead of an answer, not a correct answer followed by invented turns."}]},"prev":{"slug":"downloading-models-properly","title":"Downloading models without wasting a day","url":"https://addaly.com/learn/running-models-yourself/downloading-models-properly"},"next":{"slug":"context-settings-that-truncate-in-silence","title":"Context settings that truncate in silence","url":"https://addaly.com/learn/running-models-yourself/context-settings-that-truncate-in-silence"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Chat templates: the most common silent failure', in Running Models Yourself. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/running-models-yourself/chat-templates (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Chat templates: the most common silent failure","courseTitle":"Running Models Yourself","module":{"index":4,"slug":"getting-it-running","title":"Getting it running"},"answer":null,"keyIdea":"The model receives one formatted string, not a message list, and a wrong template or a missing stop token produces exactly the symptoms people blame on the weights: print the actual prompt before concluding anything.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"type":"stack","title":"Four ways a template fails without raising an error","layers":[{"label":"A borrowed template","detail":"A community quant packaged with another family’s markers"},{"label":"A doubled beginning-of-sequence token","detail":"The template emits one and the tokeniser adds another"},{"label":"A system role the model never had","detail":"Your system prompt is folded into the user turn, or dropped"},{"label":"The end marker missing from the stop list","detail":"It answers, then invents the next user turn and answers that"}],"caption":"The model receives one formatted string, never a list of messages. Print the string before concluding anything about the weights: every failure here is visible in it, none of them raises an error, and together they account for a large share of reports that a model is stupid."},"alt":"Figure: Four ways a template fails without raising an error. Layers, top to bottom: A borrowed template (A community quant packaged with another family’s markers); A doubled beginning-of-sequence token (The template emits one and the tokeniser adds another); A system role the model never had (Your system prompt is folded into the user turn, or dropped); The end marker missing from the stop list (It answers, then invents the next user turn and answers that). The model receives one formatted string, never a list of messages. Print the string before concluding anything about the weights: every failure here is visible in it, none of them raises an error, and together they account for a large share of reports that a model is stupid."}},"blocks":[{"t":"h2","id":"the-model-never-sees-your-messages","runs":[{"text":"The model never sees your messages"}]},{"t":"p","runs":[{"text":"A chat model does not receive a list of messages. It receives one string of tokens. Something has to convert "},{"text":"[{\"role\": \"user\", \"content\": \"hello\"}]","code":true},{"text":" into that string, and the conversion has to match the format the model was fine-tuned on, exactly."}]},{"t":"p","runs":[{"text":"That conversion is the chat template. When it is wrong, the model is being addressed in a dialect it does not recognise. It still answers, models are robust, but noticeably worse: ignoring the system prompt, failing to stop, drifting out of character, or producing rambling text. This accounts for a large share of “this model is stupid” reports, and it is invisible unless you look."}]},{"t":"h2","id":"what-the-string-looks-like","runs":[{"text":"What the string looks like"}]},{"t":"p","runs":[{"text":"A typical modern format:"}]},{"t":"code","lang":"","label":"Text","text":"<|im_start|>system\nYou are a careful assistant.<|im_end|>\n<|im_start|>user\nWhat is the boiling point of water?<|im_end|>\n<|im_start|>assistant"},{"t":"p","runs":[{"text":"The markers are single tokens in the model’s vocabulary, learned during fine-tuning. The trailing "},{"text":"<|im_start|>assistant","code":true},{"text":" is the cue to begin answering. Different families use different markers, and the differences are not cosmetic: a model trained on one set has no idea what another’s means."}]},{"t":"h2","id":"applying-it-correctly","runs":[{"text":"Applying it correctly"}]},{"t":"p","runs":[{"text":"The template is stored with the model, so let the tokeniser apply it:"}]},{"t":"code","lang":"python","label":"Python","text":"from transformers import AutoTokenizer\ntok = AutoTokenizer.from_pretrained(\"Qwen/Qwen3-8B\")\n\nmsgs = [{\"role\": \"system\", \"content\": \"You are terse.\"},\n        {\"role\": \"user\", \"content\": \"Why is the sky blue?\"}]\n\nprint(tok.apply_chat_template(msgs, tokenize=False,\n                              add_generation_prompt=True))"},{"t":"p","runs":[{"text":"Print it. Reading the actual string once teaches you more than any explanation, and it is the fastest diagnostic when something is off."}]},{"t":"p","runs":[{"text":"add_generation_prompt=True","code":true},{"text":" appends the assistant cue. Omit it and the model has not been told it is its turn, and will often continue the user’s message instead: a bug that looks like the model misunderstanding the question."}]},{"t":"h2","id":"the-three-failures-worth-recognising","runs":[{"text":"The three failures worth recognising"}]},{"t":"p","runs":[{"text":"Double BOS.","b":true},{"text":" Many templates already emit a beginning-of-sequence token. If the tokeniser then adds another, the model starts with a sequence it has never seen. Symptoms are subtle: slightly worse quality, occasionally much worse. When calling a tokeniser directly, "},{"text":"add_special_tokens=False","code":true},{"text":" on already-templated text is the fix."}]},{"t":"p","runs":[{"text":"A borrowed template.","b":true},{"text":" A community quant packaged with a template copied from a different model family. The markers are wrong, and the model is being spoken to in a foreign dialect. Dump the metadata and compare against the original repository’s "},{"text":"tokenizer_config.json","code":true},{"text":"."}]},{"t":"p","runs":[{"text":"A system role the model does not have.","b":true},{"text":" Some models were trained without a system role. Passing one gets it silently folded into the first user message, or dropped. If your system prompt seems to have no effect, check whether the template actually places it anywhere."}]},{"t":"h2","id":"seeing-what-your-server-sends","runs":[{"text":"Seeing what your server sends"}]},{"t":"p","runs":[{"text":"If you use an OpenAI-compatible endpoint, the server applies the template for you and you never see the result. Two ways to look:"}]},{"t":"code","lang":"bash","label":"Bash","text":"# llama-server logs the formatted prompt at higher verbosity\nllama-server -m model.gguf --verbose\n\n# or bypass chat entirely and send the raw string yourself\ncurl http://localhost:8080/v1/completions -H 'Content-Type: application/json' \\\n  -d '{\"prompt\":\"<|im_start|>user\\nhello<|im_end|>\\n<|im_start|>assistant\\n\",\n       \"max_tokens\":64}'"},{"t":"p","runs":[{"text":"The second is the diagnostic that settles it. If the raw call behaves correctly and the chat call does not, the template is your problem."}]},{"t":"h2","id":"overriding-a-template","runs":[{"text":"Overriding a template"}]},{"t":"p","runs":[{"text":"When the packaged template is wrong, replace it rather than working around it:"}]},{"t":"code","lang":"bash","label":"Bash","text":"llama-server -m model.gguf --chat-template chatml\n# or supply a Jinja file\nllama-server -m model.gguf --chat-template-file ./correct.jinja"},{"t":"p","runs":[{"text":"In Ollama the template lives in the Modelfile, and "},{"text":"ollama show <model> --modelfile","code":true},{"text":" prints it. Copy it, correct it, and rebuild the model with "},{"text":"ollama create","code":true},{"text":"."}]},{"t":"h2","id":"stop-tokens","runs":[{"text":"Stop tokens"}]},{"t":"p","runs":[{"text":"The other half of the same problem. The template’s end marker must be in the server’s stop list, or generation runs past the end of the answer and the model begins writing the next turn of the conversation itself: inventing a user question and answering it. If your model will not shut up, this is why, and the fix is a stop sequence rather than a shorter "},{"text":"max_tokens","code":true},{"text":"."}]},{"t":"p","runs":[{"text":"Good engines read the stop tokens from the model’s metadata. When a model is new, or a quant was packaged carelessly, they are missing and you must supply them:"}]},{"t":"code","lang":"json","label":"JSON","text":"{\"stop\": [\"<|im_end|>\", \"<|endoftext|>\"]}"},{"t":"p","runs":[{"text":"Check this on the first run of any new model. It takes one prompt and it saves a long, confused investigation."}]}],"check":{"question":[{"text":"A model answers a question correctly, then writes a plausible follow-up question from the user and answers that too, continuing until it hits the token limit. What is the mechanism?"}],"options":[{"text":[{"text":"The repetition penalty is too low, so the model is not discouraged from re-entering the conversational pattern it just produced."}],"correct":false,"tempting":[{"text":"Applies a sampler remedy to a termination problem; penalties change token probabilities, not whether generation stops at a boundary."}]},{"text":[{"text":"The context window is larger than the model’s trained length, so it continues generating to fill the space it was given."}],"correct":false,"tempting":[{"text":"Treats the window as a target to fill, which it is not; models stop at their end token regardless of remaining space."}]},{"text":[{"text":"The end-of-turn marker is not registered as a stop sequence, so generation continues past it and the model simply predicts the next thing that would follow: another turn."}],"correct":true,"tempting":null},{"text":[{"text":"The chat template omitted the generation prompt, so the model is completing the user’s message rather than replying."}],"correct":false,"tempting":[{"text":"Names a real template bug with a different signature: that failure produces a continuation instead of an answer, not a correct answer followed by invented turns."}]}],"order":[1,2,3,0],"why":null},"prev":{"slug":"downloading-models-properly","title":"Downloading models without wasting a day","minutes":8},"next":{"slug":"context-settings-that-truncate-in-silence","title":"Context settings that truncate in silence","minutes":9}}}