{"course":{"slug":"hugging-face","title":"Hugging Face, End to End","url":"https://addaly.com/learn/hugging-face","level":"intermediate","track":"create"},"lesson":{"slug":"chat-templates","title":"The chat template is not optional","url":"https://addaly.com/learn/hugging-face/chat-templates","position":15,"of":75,"minutes":9,"answer":"An instruction-tuned model was trained on one exact string format that ships inside tokenizer_config.json, so apply_chat_template(....","keyIdea":"An instruction-tuned model was trained on one exact string format that ships inside `tokenizer_config.json`, so `apply_chat_template(..., add_generation_prompt=True)` is the only reliable way to build a prompt: hand-written `User:`/`Assistant:` text puts the model outside its training distribution with no error to show for it."},"body":{"format":"markdown","text":"## The most common quality bug in open models\n\nSomebody downloads an instruct model, writes this, and concludes it is weak:\n\n```python\nprompt = \"User: Summarise this in two lines.\\n\\nAssistant:\"\n```\n\nThat is not how the model was trained. During instruction tuning it saw a specific string (specific role markers, specific special tokens, specific newlines) thousands of times. Give it something that looks similar but is not identical and you are outside the distribution it learned. Output gets vaguer, it ignores system instructions, it fails to stop, it starts inventing a second user turn.\n\nThe correct format is not a convention you can guess. It is stored in the repository, in `tokenizer_config.json`, as a Jinja string named `chat_template`.\n\n> Figure: Two prompts the model sees very differently. Typed by hand: User: and Assistant: written as plain text; None of the special tokens the model was trained on; Nothing marks where the assistant's turn begins; The model often invents a second user turn; Rewritten by hand for every model family. Rendered by apply_chat_template: The exact Jinja string from tokenizer_config.json; The role markers the model saw thousands of times; add_generation_prompt opens the assistant turn; The end marker matches the configured eos_token_id; Unchanged when you swap the model underneath. Neither raises an error and both produce fluent text, which is why this is the commonest quality bug in open models. The left column is a string that resembles the training format; the right is the training format, read out of the repository the weights came from.\n\n## Use it\n\n```python\nmessages = [\n    {\"role\": \"system\", \"content\": \"You reply in one sentence.\"},\n    {\"role\": \"user\", \"content\": \"Why is the sky blue?\"},\n]\n\ntext = tok.apply_chat_template(\n    messages,\n    tokenize=False,\n    add_generation_prompt=True,\n)\nprint(repr(text))\n```\n\nFor a ChatML-style model that prints something like:\n\n```\n<|im_start|>system\nYou reply in one sentence.<|im_end|>\n<|im_start|>user\nWhy is the sky blue?<|im_end|>\n<|im_start|>assistant\n```\n\nEvery model family differs. Llama 3 uses `<|start_header_id|>` markers, Mistral uses `[INST]` and `[/INST]`, Gemma uses `<start_of_turn>`. Guessing wrong is invisible in the output shape and visible in the quality.\n\n`add_generation_prompt=True` is the part people miss. It appends the opening marker for the assistant's turn, so the model knows it is its turn to speak. Without it the model may produce another user message, because that is what would plausibly come next.\n\n## Print it once per model\n\n`tokenize=False` exists so you can look. Do it once for every new model: it takes five seconds and tells you the system-role support, the turn markers, and the stop token:\n\n```python\nprint(tok.chat_template)          # the Jinja source\nprint(tok.eos_token, tok.eos_token_id)\n```\n\nSome models have no system role, and their template silently folds the system message into the first user turn, or drops it. That explains a whole class of \"it ignores my instructions\" reports.\n\n## In the pipeline\n\nThe text-generation pipeline applies the template for you when you pass a list of messages rather than a string:\n\n```python\npipe = pipeline(\"text-generation\", model=\"Qwen/Qwen2.5-1.5B-Instruct\")\nout = pipe(messages, max_new_tokens=128)\nprint(out[0][\"generated_text\"][-1][\"content\"])\n```\n\nThat is the recommended path, and it is why the message-list form is worth adopting even in throwaway scripts: it stops you hand-writing a format you will get wrong later.\n\n## Multi-turn, and the thing to get right\n\nA conversation is the whole list, re-templated every turn:\n\n```python\nmessages.append({\"role\": \"assistant\", \"content\": reply})\nmessages.append({\"role\": \"user\", \"content\": follow_up})\n```\n\nAppend the assistant's *clean* reply, not the raw decode with special tokens still in it. Feeding `<|im_end|>` back in as message content corrupts the next template render, and the model starts emitting stray markers.\n\n## Tool calling lives here too\n\nModels trained to call functions define the format in the same template. `apply_chat_template(messages, tools=[...])` renders the tool schemas into whatever shape that family expects, and the model's tool calls come back as text you parse. There is no cross-model standard, which is why a tool-calling application written against one model needs work to move to another. Frameworks paper over this; the underlying variation is real.\n\n## Two models, one application\n\nIf your code has to support more than one model, keep the messages list as the only representation of a conversation anywhere in your program, and render it at the last possible moment with whichever tokenizer is loaded. The moment a rendered string with `[INST]` in it is stored in a database or passed between functions, you have hard-coded one family, and swapping the model becomes a refactor rather than a config change.\n\nThe same rule protects your evaluation set. Store the messages, not the prompt. A set of rendered prompts measures one model and cannot be reused against its successor, which is the quiet way a team ends up unable to compare anything.\n\n## When there is no template\n\nBase models have none, and `apply_chat_template` raises or falls back to a default. That error is useful information: it usually means you picked a base checkpoint when you wanted an instruct one. Occasionally a genuine instruct model ships without its template by oversight; then the card, or a discussion thread, is where the format is written, and it is worth opening a pull request to add it.\n\n## Do this now\n\nPrint the rendered template for two instruct models from different families with the same two messages. Put the outputs side by side. They will share no markers at all; and that is the whole argument for never writing the prompt format by hand."},"check":{"question":"A model keeps writing a plausible follow-up question from the user instead of answering. The messages list and template are otherwise correct. What is the likely omission?","options":[{"text":"`add_generation_prompt=True` was not set, so the rendered string does not open the assistant's turn.","correct":true},{"text":"The system message is missing, and without one the model has no instruction to answer rather than converse.","correct":false,"tempting":"A system message shapes style, but its absence does not make a model impersonate the user."},{"text":"The eos token is absent from the template, so generation runs past the end of the answer into invented turns.","correct":false,"tempting":"Would produce an answer followed by extra turns, rather than no answer at all."},{"text":"The previous assistant reply was appended with its special tokens intact, which corrupts the role sequence in the render.","correct":false,"tempting":"A genuine bug with a similar smell, but it produces stray markers in output rather than a clean user-style turn on a first message."}]},"prev":{"slug":"from-logits-to-text","title":"Logits, and the four knobs between them and words","url":"https://addaly.com/learn/hugging-face/from-logits-to-text"},"next":{"slug":"batching-and-throughput","title":"Throughput and latency are different problems","url":"https://addaly.com/learn/hugging-face/batching-and-throughput"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'The chat template is not optional', in Hugging Face, End to End. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/hugging-face/chat-templates (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"The chat template is not optional","courseTitle":"Hugging Face, End to End","module":{"index":2,"slug":"transformers-from-text-to-text","title":"Transformers, from characters to output"},"answer":null,"keyIdea":"An instruction-tuned model was trained on one exact string format that ships inside `tokenizer_config.json`, so `apply_chat_template(..., add_generation_prompt=True)` is the only reliable way to build a prompt: hand-written `User:`/`Assistant:` text puts the model outside its training distribution with no error to show for it.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"left":{"title":"Typed by hand","items":["User: and Assistant: written as plain text","None of the special tokens the model was trained on","Nothing marks where the assistant’s turn begins","The model often invents a second user turn","Rewritten by hand for every model family"]},"type":"compare","right":{"title":"Rendered by apply_chat_template","items":["The exact Jinja string from tokenizer_config.json","The role markers the model saw thousands of times","add_generation_prompt opens the assistant turn","The end marker matches the configured eos_token_id","Unchanged when you swap the model underneath"]},"title":"Two prompts the model sees very differently","caption":"Neither raises an error and both produce fluent text, which is why this is the commonest quality bug in open models. The left column is a string that resembles the training format; the right is the training format, read out of the repository the weights came from."},"alt":"Figure: Two prompts the model sees very differently. Typed by hand: User: and Assistant: written as plain text; None of the special tokens the model was trained on; Nothing marks where the assistant’s turn begins; The model often invents a second user turn; Rewritten by hand for every model family. Rendered by apply_chat_template: The exact Jinja string from tokenizer_config.json; The role markers the model saw thousands of times; add_generation_prompt opens the assistant turn; The end marker matches the configured eos_token_id; Unchanged when you swap the model underneath. Neither raises an error and both produce fluent text, which is why this is the commonest quality bug in open models. The left column is a string that resembles the training format; the right is the training format, read out of the repository the weights came from."}},"blocks":[{"t":"h2","id":"the-most-common-quality-bug-in-open-models","runs":[{"text":"The most common quality bug in open models"}]},{"t":"p","runs":[{"text":"Somebody downloads an instruct model, writes this, and concludes it is weak:"}]},{"t":"code","lang":"python","label":"Python","text":"prompt = \"User: Summarise this in two lines.\\n\\nAssistant:\""},{"t":"p","runs":[{"text":"That is not how the model was trained. During instruction tuning it saw a specific string (specific role markers, specific special tokens, specific newlines) thousands of times. Give it something that looks similar but is not identical and you are outside the distribution it learned. Output gets vaguer, it ignores system instructions, it fails to stop, it starts inventing a second user turn."}]},{"t":"p","runs":[{"text":"The correct format is not a convention you can guess. It is stored in the repository, in "},{"text":"tokenizer_config.json","code":true},{"text":", as a Jinja string named "},{"text":"chat_template","code":true},{"text":"."}]},{"t":"h2","id":"use-it","runs":[{"text":"Use it"}]},{"t":"code","lang":"python","label":"Python","text":"messages = [\n    {\"role\": \"system\", \"content\": \"You reply in one sentence.\"},\n    {\"role\": \"user\", \"content\": \"Why is the sky blue?\"},\n]\n\ntext = tok.apply_chat_template(\n    messages,\n    tokenize=False,\n    add_generation_prompt=True,\n)\nprint(repr(text))"},{"t":"p","runs":[{"text":"For a ChatML-style model that prints something like:"}]},{"t":"code","lang":"","label":"Text","text":"<|im_start|>system\nYou reply in one sentence.<|im_end|>\n<|im_start|>user\nWhy is the sky blue?<|im_end|>\n<|im_start|>assistant"},{"t":"p","runs":[{"text":"Every model family differs. Llama 3 uses "},{"text":"<|start_header_id|>","code":true},{"text":" markers, Mistral uses "},{"text":"[INST]","code":true},{"text":" and "},{"text":"[/INST]","code":true},{"text":", Gemma uses "},{"text":"<start_of_turn>","code":true},{"text":". Guessing wrong is invisible in the output shape and visible in the quality."}]},{"t":"p","runs":[{"text":"add_generation_prompt=True","code":true},{"text":" is the part people miss. It appends the opening marker for the assistant’s turn, so the model knows it is its turn to speak. Without it the model may produce another user message, because that is what would plausibly come next."}]},{"t":"h2","id":"print-it-once-per-model","runs":[{"text":"Print it once per model"}]},{"t":"p","runs":[{"text":"tokenize=False","code":true},{"text":" exists so you can look. Do it once for every new model: it takes five seconds and tells you the system-role support, the turn markers, and the stop token:"}]},{"t":"code","lang":"python","label":"Python","text":"print(tok.chat_template)          # the Jinja source\nprint(tok.eos_token, tok.eos_token_id)"},{"t":"p","runs":[{"text":"Some models have no system role, and their template silently folds the system message into the first user turn, or drops it. That explains a whole class of “it ignores my instructions” reports."}]},{"t":"h2","id":"in-the-pipeline","runs":[{"text":"In the pipeline"}]},{"t":"p","runs":[{"text":"The text-generation pipeline applies the template for you when you pass a list of messages rather than a string:"}]},{"t":"code","lang":"python","label":"Python","text":"pipe = pipeline(\"text-generation\", model=\"Qwen/Qwen2.5-1.5B-Instruct\")\nout = pipe(messages, max_new_tokens=128)\nprint(out[0][\"generated_text\"][-1][\"content\"])"},{"t":"p","runs":[{"text":"That is the recommended path, and it is why the message-list form is worth adopting even in throwaway scripts: it stops you hand-writing a format you will get wrong later."}]},{"t":"h2","id":"multi-turn-and-the-thing-to-get-right","runs":[{"text":"Multi-turn, and the thing to get right"}]},{"t":"p","runs":[{"text":"A conversation is the whole list, re-templated every turn:"}]},{"t":"code","lang":"python","label":"Python","text":"messages.append({\"role\": \"assistant\", \"content\": reply})\nmessages.append({\"role\": \"user\", \"content\": follow_up})"},{"t":"p","runs":[{"text":"Append the assistant’s "},{"text":"clean","i":true},{"text":" reply, not the raw decode with special tokens still in it. Feeding "},{"text":"<|im_end|>","code":true},{"text":" back in as message content corrupts the next template render, and the model starts emitting stray markers."}]},{"t":"h2","id":"tool-calling-lives-here-too","runs":[{"text":"Tool calling lives here too"}]},{"t":"p","runs":[{"text":"Models trained to call functions define the format in the same template. "},{"text":"apply_chat_template(messages, tools=[...])","code":true},{"text":" renders the tool schemas into whatever shape that family expects, and the model’s tool calls come back as text you parse. There is no cross-model standard, which is why a tool-calling application written against one model needs work to move to another. Frameworks paper over this; the underlying variation is real."}]},{"t":"h2","id":"two-models-one-application","runs":[{"text":"Two models, one application"}]},{"t":"p","runs":[{"text":"If your code has to support more than one model, keep the messages list as the only representation of a conversation anywhere in your program, and render it at the last possible moment with whichever tokenizer is loaded. The moment a rendered string with "},{"text":"[INST]","code":true},{"text":" in it is stored in a database or passed between functions, you have hard-coded one family, and swapping the model becomes a refactor rather than a config change."}]},{"t":"p","runs":[{"text":"The same rule protects your evaluation set. Store the messages, not the prompt. A set of rendered prompts measures one model and cannot be reused against its successor, which is the quiet way a team ends up unable to compare anything."}]},{"t":"h2","id":"when-there-is-no-template","runs":[{"text":"When there is no template"}]},{"t":"p","runs":[{"text":"Base models have none, and "},{"text":"apply_chat_template","code":true},{"text":" raises or falls back to a default. That error is useful information: it usually means you picked a base checkpoint when you wanted an instruct one. Occasionally a genuine instruct model ships without its template by oversight; then the card, or a discussion thread, is where the format is written, and it is worth opening a pull request to add it."}]},{"t":"h2","id":"do-this-now","runs":[{"text":"Do this now"}],"role":"try"},{"t":"p","runs":[{"text":"Print the rendered template for two instruct models from different families with the same two messages. Put the outputs side by side. They will share no markers at all; and that is the whole argument for never writing the prompt format by hand."}]}],"check":{"question":[{"text":"A model keeps writing a plausible follow-up question from the user instead of answering. The messages list and template are otherwise correct. What is the likely omission?"}],"options":[{"text":[{"text":"add_generation_prompt=True","code":true},{"text":" was not set, so the rendered string does not open the assistant’s turn."}],"correct":true,"tempting":null},{"text":[{"text":"The system message is missing, and without one the model has no instruction to answer rather than converse."}],"correct":false,"tempting":[{"text":"A system message shapes style, but its absence does not make a model impersonate the user."}]},{"text":[{"text":"The eos token is absent from the template, so generation runs past the end of the answer into invented turns."}],"correct":false,"tempting":[{"text":"Would produce an answer followed by extra turns, rather than no answer at all."}]},{"text":[{"text":"The previous assistant reply was appended with its special tokens intact, which corrupts the role sequence in the render."}],"correct":false,"tempting":[{"text":"A genuine bug with a similar smell, but it produces stray markers in output rather than a clean user-style turn on a first message."}]}],"order":[1,0,3,2],"why":null},"prev":{"slug":"from-logits-to-text","title":"Logits, and the four knobs between them and words","minutes":10},"next":{"slug":"batching-and-throughput","title":"Throughput and latency are different problems","minutes":9}}}