{"course":{"slug":"open-models","title":"The Open Model Ecosystem","url":"https://addaly.com/learn/open-models","level":"intermediate","track":"deep"},"lesson":{"slug":"the-chat-template-contract","title":"The chat template is a contract","url":"https://addaly.com/learn/open-models/the-chat-template-contract","position":24,"of":71,"minutes":9,"answer":"An instruct model was trained on an exact token sequence that ships with it in tokenizer_config.json, so build prompts with apply_chat_template and add_generation_prompt rather than by hand, and watch for a duplicated beginning-of-sequence token.","keyIdea":"An instruct model was trained on an exact token sequence that ships with it in tokenizer_config.json, so build prompts with apply_chat_template and add_generation_prompt rather than by hand, and watch for a duplicated beginning-of-sequence token."},"body":{"format":"markdown","text":"## The most common self-inflicted wound\n\nA team tries an open model, finds it rambles, ignores the system prompt and never stops, and concludes the model is poor. Nine times in ten the model is fine and the conversation was formatted wrongly. This lesson is how to check in thirty seconds.\n\nAn instruct model was trained on an exact byte sequence: special tokens marking where each role starts and ends, a specific ordering, a specific token that means the assistant should now speak. Deviate and you are sampling from a distribution the model was never trained on.\n\n## Where the template lives\n\nInside `tokenizer_config.json`, in a field called `chat_template`, written in Jinja. It ships with the model, which means the model tells you its own format and you never have to guess:\n\n```python\nfrom transformers import AutoTokenizer\ntok = AutoTokenizer.from_pretrained(\"Qwen/Qwen3-8B\")\nprint(tok.chat_template[:400])          # the source\nprint(tok.apply_chat_template(\n    [{\"role\": \"system\", \"content\": \"You are terse.\"},\n     {\"role\": \"user\",   \"content\": \"Summarise TCP in two lines.\"}],\n    tokenize=False, add_generation_prompt=True))\n```\n\nPrint that once for every model you use. It takes ten seconds and it removes an entire category of mystery.\n\nFor a Llama 3 model the output looks like this, and every one of those markers matters:\n\n```\n<|begin_of_text|><|start_header_id|>system<|end_header_id|>\n\nYou are terse.<|eot_id|><|start_header_id|>user<|end_header_id|>\n\nSummarise TCP in two lines.<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n\n```\n\n## The four things that go wrong\n\n**Forgetting `add_generation_prompt=True`.** Without it the string ends after the user's turn and the model has not been told it is the assistant's turn. It will often continue by writing another user message. This is the single most frequent cause of a model that talks to itself.\n\n**Double beginning-of-sequence tokens.** The template already inserts `<|begin_of_text|>`. If you then call `tok(formatted_string)` with the default `add_special_tokens=True`, the tokenizer adds another. Two of them at the start measurably degrades output on some models and produces no error at all. Either use `apply_chat_template(tokenize=True)` and let it do both jobs, or pass `add_special_tokens=False` when tokenising a string the template already built.\n\n**Assuming a system role exists.** Gemma's template has no system role. Passing one either raises an exception or is silently folded into the first user turn depending on the version. Several other models accept a system message but were tuned with almost none, and follow it weakly. Read the template: if `system` does not appear in the Jinja source, the model does not have one.\n\n**Hand-writing the format from a blog post.** Formats change between versions of the same family. Llama 2 used `[INST]` markers; Llama 3 uses header tokens; they share nothing. The template in the repository is correct by construction; a remembered format is not.\n\n## Why the chat endpoint is the right default\n\nWhen you serve through llama.cpp's server, vLLM or Ollama and call `/v1/chat/completions`, the server reads the template out of the model's own metadata and applies it. That is a strong reason to use the chat endpoint rather than raw completions even when you are running locally and it feels like an unnecessary layer.\n\nThe exception is when you deliberately want no template: few-shot classification against a base model, perplexity measurement, or fill-in-the-middle code completion. Then use the completions endpoint and construct the string yourself, knowingly.\n\n## Tool calling lives here too\n\nModels that support function calling encode tools in the template, usually as a JSON block in a system turn or a dedicated tools section, with a specific token or format for the model's call. `apply_chat_template` accepts a `tools=` argument and will format them the way the model expects.\n\nThis is why tool calling that works on one model breaks on another with identical code: the format is per model, and a serving layer that does not implement that model's variant will produce calls the model was not trained to emit.\n\n## The thirty-second diagnostic\n\nWhen output is bad, before changing anything else:\n\n1. Print the exact string being sent, including special tokens.\n2. Compare it to `apply_chat_template` output for the same messages.\n3. Check for a duplicated start token at the beginning.\n4. Check the string ends with the assistant header, not with the user's turn.\n\nIf all four are right, the problem is somewhere else and you have spent thirty seconds ruling out the likeliest cause."},"check":{"question":"A locally served model keeps writing a plausible user question after answering, then answering that too. The chat template is applied correctly otherwise. What is the most likely omission?","options":[{"text":"The end-of-sequence token is missing from the tokenizer, so generation cannot terminate.","correct":false,"tempting":"Would cause generation to run to the token limit mid-sentence, not to produce a well-formed new user turn."},{"text":"add_generation_prompt was not set, so the prompt ends after the user turn and the model continues the transcript rather than replying as the assistant.","correct":true},{"text":"The system prompt is too weak and needs an explicit instruction not to invent questions.","correct":false,"tempting":"Patches a formatting fault with prompt text; the model is completing a document because it was never told the assistant turn had begun."},{"text":"Temperature is too high, so the model wanders past the end of its answer.","correct":false,"tempting":"Sampling changes word choice, not whether the model believes the transcript has ended."}]},"prev":{"slug":"tokenizers-across-families","title":"Why the same sentence costs different amounts","url":"https://addaly.com/learn/open-models/tokenizers-across-families"},"next":{"slug":"attention-and-the-kv-cache","title":"Attention variants, and the cache that eats your memory","url":"https://addaly.com/learn/open-models/attention-and-the-kv-cache"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'The chat template is a contract', in The Open Model Ecosystem. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/open-models/the-chat-template-contract (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"The chat template is a contract","courseTitle":"The Open Model Ecosystem","module":{"index":3,"slug":"inside-the-artefact","title":"Inside the files"},"answer":null,"keyIdea":"An instruct model was trained on an exact token sequence that ships with it in tokenizer_config.json, so build prompts with apply_chat_template and add_generation_prompt rather than by hand, and watch for a duplicated beginning-of-sequence token.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"the-most-common-self-inflicted-wound","runs":[{"text":"The most common self-inflicted wound"}]},{"t":"p","runs":[{"text":"A team tries an open model, finds it rambles, ignores the system prompt and never stops, and concludes the model is poor. Nine times in ten the model is fine and the conversation was formatted wrongly. This lesson is how to check in thirty seconds."}]},{"t":"p","runs":[{"text":"An instruct model was trained on an exact byte sequence: special tokens marking where each role starts and ends, a specific ordering, a specific token that means the assistant should now speak. Deviate and you are sampling from a distribution the model was never trained on."}]},{"t":"h2","id":"where-the-template-lives","runs":[{"text":"Where the template lives"}]},{"t":"p","runs":[{"text":"Inside "},{"text":"tokenizer_config.json","code":true},{"text":", in a field called "},{"text":"chat_template","code":true},{"text":", written in Jinja. It ships with the model, which means the model tells you its own format and you never have to guess:"}]},{"t":"code","lang":"python","label":"Python","text":"from transformers import AutoTokenizer\ntok = AutoTokenizer.from_pretrained(\"Qwen/Qwen3-8B\")\nprint(tok.chat_template[:400])          # the source\nprint(tok.apply_chat_template(\n    [{\"role\": \"system\", \"content\": \"You are terse.\"},\n     {\"role\": \"user\",   \"content\": \"Summarise TCP in two lines.\"}],\n    tokenize=False, add_generation_prompt=True))"},{"t":"p","runs":[{"text":"Print that once for every model you use. It takes ten seconds and it removes an entire category of mystery."}]},{"t":"p","runs":[{"text":"For a Llama 3 model the output looks like this, and every one of those markers matters:"}]},{"t":"code","lang":"","label":"Text","text":"<|begin_of_text|><|start_header_id|>system<|end_header_id|>\n\nYou are terse.<|eot_id|><|start_header_id|>user<|end_header_id|>\n\nSummarise TCP in two lines.<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n"},{"t":"h2","id":"the-four-things-that-go-wrong","runs":[{"text":"The four things that go wrong"}]},{"t":"p","runs":[{"text":"Forgetting ","b":true},{"text":"add_generation_prompt=True","b":true,"code":true},{"text":".","b":true},{"text":" Without it the string ends after the user’s turn and the model has not been told it is the assistant’s turn. It will often continue by writing another user message. This is the single most frequent cause of a model that talks to itself."}]},{"t":"p","runs":[{"text":"Double beginning-of-sequence tokens.","b":true},{"text":" The template already inserts "},{"text":"<|begin_of_text|>","code":true},{"text":". If you then call "},{"text":"tok(formatted_string)","code":true},{"text":" with the default "},{"text":"add_special_tokens=True","code":true},{"text":", the tokenizer adds another. Two of them at the start measurably degrades output on some models and produces no error at all. Either use "},{"text":"apply_chat_template(tokenize=True)","code":true},{"text":" and let it do both jobs, or pass "},{"text":"add_special_tokens=False","code":true},{"text":" when tokenising a string the template already built."}]},{"t":"p","runs":[{"text":"Assuming a system role exists.","b":true},{"text":" Gemma’s template has no system role. Passing one either raises an exception or is silently folded into the first user turn depending on the version. Several other models accept a system message but were tuned with almost none, and follow it weakly. Read the template: if "},{"text":"system","code":true},{"text":" does not appear in the Jinja source, the model does not have one."}]},{"t":"p","runs":[{"text":"Hand-writing the format from a blog post.","b":true},{"text":" Formats change between versions of the same family. Llama 2 used "},{"text":"[INST]","code":true},{"text":" markers; Llama 3 uses header tokens; they share nothing. The template in the repository is correct by construction; a remembered format is not."}]},{"t":"h2","id":"why-the-chat-endpoint-is-the-right-default","runs":[{"text":"Why the chat endpoint is the right default"}]},{"t":"p","runs":[{"text":"When you serve through llama.cpp’s server, vLLM or Ollama and call "},{"text":"/v1/chat/completions","code":true},{"text":", the server reads the template out of the model’s own metadata and applies it. That is a strong reason to use the chat endpoint rather than raw completions even when you are running locally and it feels like an unnecessary layer."}]},{"t":"p","runs":[{"text":"The exception is when you deliberately want no template: few-shot classification against a base model, perplexity measurement, or fill-in-the-middle code completion. Then use the completions endpoint and construct the string yourself, knowingly."}]},{"t":"h2","id":"tool-calling-lives-here-too","runs":[{"text":"Tool calling lives here too"}]},{"t":"p","runs":[{"text":"Models that support function calling encode tools in the template, usually as a JSON block in a system turn or a dedicated tools section, with a specific token or format for the model’s call. "},{"text":"apply_chat_template","code":true},{"text":" accepts a "},{"text":"tools=","code":true},{"text":" argument and will format them the way the model expects."}]},{"t":"p","runs":[{"text":"This is why tool calling that works on one model breaks on another with identical code: the format is per model, and a serving layer that does not implement that model’s variant will produce calls the model was not trained to emit."}]},{"t":"h2","id":"the-thirty-second-diagnostic","runs":[{"text":"The thirty-second diagnostic"}]},{"t":"p","runs":[{"text":"When output is bad, before changing anything else:"}]},{"t":"ol","start":1,"items":[[{"text":"Print the exact string being sent, including special tokens."}],[{"text":"Compare it to "},{"text":"apply_chat_template","code":true},{"text":" output for the same messages."}],[{"text":"Check for a duplicated start token at the beginning."}],[{"text":"Check the string ends with the assistant header, not with the user’s turn."}]]},{"t":"p","runs":[{"text":"If all four are right, the problem is somewhere else and you have spent thirty seconds ruling out the likeliest cause."}]}],"check":{"question":[{"text":"A locally served model keeps writing a plausible user question after answering, then answering that too. The chat template is applied correctly otherwise. What is the most likely omission?"}],"options":[{"text":[{"text":"The end-of-sequence token is missing from the tokenizer, so generation cannot terminate."}],"correct":false,"tempting":[{"text":"Would cause generation to run to the token limit mid-sentence, not to produce a well-formed new user turn."}]},{"text":[{"text":"add_generation_prompt was not set, so the prompt ends after the user turn and the model continues the transcript rather than replying as the assistant."}],"correct":true,"tempting":null},{"text":[{"text":"The system prompt is too weak and needs an explicit instruction not to invent questions."}],"correct":false,"tempting":[{"text":"Patches a formatting fault with prompt text; the model is completing a document because it was never told the assistant turn had begun."}]},{"text":[{"text":"Temperature is too high, so the model wanders past the end of its answer."}],"correct":false,"tempting":[{"text":"Sampling changes word choice, not whether the model believes the transcript has ended."}]}],"order":[2,1,0,3],"why":null},"prev":{"slug":"tokenizers-across-families","title":"Why the same sentence costs different amounts","minutes":9},"next":{"slug":"attention-and-the-kv-cache","title":"Attention variants, and the cache that eats your memory","minutes":10}}}