{"course":{"slug":"how-llms-work","title":"How a Language Model Actually Works","url":"https://addaly.com/learn/how-llms-work","level":"intermediate","track":"start"},"lesson":{"slug":"stop-tokens-and-chat-templates","title":"Special tokens, stop conditions, and where a turn ends","url":"https://addaly.com/learn/how-llms-work/stop-tokens-and-chat-templates","position":24,"of":78,"minutes":8,"answer":"A conversation is one long string containing learned boundary markers, and most \"the model went haywire\" reports are a boundary marker that was missing or wrong.","keyIdea":"A conversation is one long string containing learned boundary markers, and most \"the model went haywire\" reports are a boundary marker that was missing or wrong."},"body":{"format":"markdown","text":"## Nothing tells the model the turn is over\n\nGeneration runs until something stops it. There are exactly three stopping conditions, and they are all mundane:\n\n1. The model emits its **end-of-turn token**, and the serving code recognises it and halts.\n2. The generated text matches a **stop sequence** you supplied.\n3. The **maximum token count** is reached.\n\nThat is all. There is no notion of a complete thought. A response that ends mid-sentence hit condition 3, and the fix is a larger limit, not a better prompt.\n\n## Special tokens are ordinary integers\n\nThe markers around roles (written variously as `<|im_start|>`, `<|eot_id|>`, `<|end|>`) are entries in the vocabulary like any other. They have ids, embeddings, and logits. What makes them special is only that training data was formatted with them, so the model learned to emit an end-of-turn token where an assistant reply naturally finishes.\n\nCheck `generation_config.json` in any open model and you will find `eos_token_id`. Some models have several valid stop tokens, and a serving stack configured with only one will produce output that runs on and on, often with the model helpfully inventing the user's next message. That symptom is diagnostic: **if your model writes both sides of the conversation, your stop token configuration is wrong.**\n\n## Why the template must be exact\n\nModule one showed the chat template lives in `tokenizer_config.json`. Two failure modes follow from getting it wrong, and both are common:\n\n**Wrong markers.** Use one family's template with another family's model and the model sees a formatting it never trained on. It still produces text, but the assistant behaviour it learned is keyed to those markers, so quality degrades in a way that looks like the model being poor rather than misconfigured.\n\n**Missing generation prompt.** After the last user message, the template must add the opening marker for the assistant turn. Omit it and the model is being asked to continue the *user's* message, so it writes another user question. In Hugging Face this is `add_generation_prompt=True`, and forgetting it is a rite of passage.\n\n```python\nprompt = tokenizer.apply_chat_template(\n    messages, tokenize=False, add_generation_prompt=True\n)\n```\n\n## Stop sequences, and the trap in them\n\nStop sequences are matched on the decoded text, not on tokens, so they can trigger mid-token in ways that look strange. Two rules:\n\n**The stop sequence is not returned.** If you stop on `\"\\n\\n\"` you get the text up to it, and you will need to add it back if your format expects it.\n\n**Beware stopping on something that can legitimately appear.** Stopping a code generator on triple backticks works until the model writes a nested code block. Stopping on `\"}\"` breaks the first time the output contains nested JSON.\n\n## The security consequence\n\nHere is the part that matters beyond configuration. Because role markers are just tokens, text arriving from outside can contain them. If your application builds a prompt by pasting in a user's message, a retrieved document, or a web page, and that text includes `<|im_start|>system`, you have handed a stranger the ability to open what looks like a system turn.\n\nMost serving stacks defend against this by refusing to tokenise special-token strings from user content: the `allowed_special` setting in some tokenizers exists exactly here. **Verify that your stack does this rather than assuming it.** The check takes ten minutes: send a message containing your model's role markers and inspect the token ids that reach the model.\n\nThis is one concrete instance of the general property that module six treats at length: there is no structural separation between instructions and data, only formatting conventions that the model learned to respect.\n\n## The five-minute diagnostic\n\nWhen a local model behaves oddly, print the exact string that goes in:\n\n```python\nprint(repr(tokenizer.apply_chat_template(messages, tokenize=False,\n                                         add_generation_prompt=True)))\n```\n\nCompare it to the model card's documented format, character by character. In my experience roughly half of \"this open model is bad\" reports are resolved at this step, and the other half by the quantisation issues in module seven."},"check":{"question":"A locally served model answers the user's question and then writes a plausible follow-up question from the user, and answers that too. What is the cause?","options":[{"text":"The temperature is too high, so the model wanders past its intended answer.","correct":false,"tempting":"High temperature does produce rambling, and turning it down is the reflexive first fix for strange output."},{"text":"The context window includes previous turns that the model is continuing.","correct":false,"tempting":"History leaking into a prompt is a real bug class, and it would explain conversational-looking output."},{"text":"The maximum token limit is set too high, letting the model keep going.","correct":false,"tempting":"The limit does bound length, so raising or lowering it feels like the obvious control over runaway output."},{"text":"The end-of-turn token is not configured as a stop condition, so nothing halts generation when the assistant's turn naturally ends.","correct":true}]},"prev":{"slug":"reading-logprobs","title":"Reading the numbers the model will give you","url":"https://addaly.com/learn/how-llms-work/reading-logprobs"},"next":{"slug":"constrained-decoding","title":"Making output structurally valid, without asking nicely","url":"https://addaly.com/learn/how-llms-work/constrained-decoding"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Special tokens, stop conditions, and where a turn ends', in How a Language Model Actually Works. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/how-llms-work/stop-tokens-and-chat-templates (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Special tokens, stop conditions, and where a turn ends","courseTitle":"How a Language Model Actually Works","module":{"index":3,"slug":"turning-scores-into-text","title":"Turning scores into text"},"answer":null,"keyIdea":"A conversation is one long string containing learned boundary markers, and most “the model went haywire” reports are a boundary marker that was missing or wrong.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"nothing-tells-the-model-the-turn-is-over","runs":[{"text":"Nothing tells the model the turn is over"}]},{"t":"p","runs":[{"text":"Generation runs until something stops it. There are exactly three stopping conditions, and they are all mundane:"}]},{"t":"ol","start":1,"items":[[{"text":"The model emits its "},{"text":"end-of-turn token","b":true},{"text":", and the serving code recognises it and halts."}],[{"text":"The generated text matches a "},{"text":"stop sequence","b":true},{"text":" you supplied."}],[{"text":"The "},{"text":"maximum token count","b":true},{"text":" is reached."}]]},{"t":"p","runs":[{"text":"That is all. There is no notion of a complete thought. A response that ends mid-sentence hit condition 3, and the fix is a larger limit, not a better prompt."}]},{"t":"h2","id":"special-tokens-are-ordinary-integers","runs":[{"text":"Special tokens are ordinary integers"}]},{"t":"p","runs":[{"text":"The markers around roles (written variously as "},{"text":"<|im_start|>","code":true},{"text":", "},{"text":"<|eot_id|>","code":true},{"text":", "},{"text":"<|end|>","code":true},{"text":") are entries in the vocabulary like any other. They have ids, embeddings, and logits. What makes them special is only that training data was formatted with them, so the model learned to emit an end-of-turn token where an assistant reply naturally finishes."}]},{"t":"p","runs":[{"text":"Check "},{"text":"generation_config.json","code":true},{"text":" in any open model and you will find "},{"text":"eos_token_id","code":true},{"text":". Some models have several valid stop tokens, and a serving stack configured with only one will produce output that runs on and on, often with the model helpfully inventing the user’s next message. That symptom is diagnostic: "},{"text":"if your model writes both sides of the conversation, your stop token configuration is wrong.","b":true}]},{"t":"h2","id":"why-the-template-must-be-exact","runs":[{"text":"Why the template must be exact"}]},{"t":"p","runs":[{"text":"Module one showed the chat template lives in "},{"text":"tokenizer_config.json","code":true},{"text":". Two failure modes follow from getting it wrong, and both are common:"}]},{"t":"p","runs":[{"text":"Wrong markers.","b":true},{"text":" Use one family’s template with another family’s model and the model sees a formatting it never trained on. It still produces text, but the assistant behaviour it learned is keyed to those markers, so quality degrades in a way that looks like the model being poor rather than misconfigured."}]},{"t":"p","runs":[{"text":"Missing generation prompt.","b":true},{"text":" After the last user message, the template must add the opening marker for the assistant turn. Omit it and the model is being asked to continue the "},{"text":"user’s","i":true},{"text":" message, so it writes another user question. In Hugging Face this is "},{"text":"add_generation_prompt=True","code":true},{"text":", and forgetting it is a rite of passage."}]},{"t":"code","lang":"python","label":"Python","text":"prompt = tokenizer.apply_chat_template(\n    messages, tokenize=False, add_generation_prompt=True\n)"},{"t":"h2","id":"stop-sequences-and-the-trap-in-them","runs":[{"text":"Stop sequences, and the trap in them"}]},{"t":"p","runs":[{"text":"Stop sequences are matched on the decoded text, not on tokens, so they can trigger mid-token in ways that look strange. Two rules:"}]},{"t":"p","runs":[{"text":"The stop sequence is not returned.","b":true},{"text":" If you stop on "},{"text":"\"\\n\\n\"","code":true},{"text":" you get the text up to it, and you will need to add it back if your format expects it."}]},{"t":"p","runs":[{"text":"Beware stopping on something that can legitimately appear.","b":true},{"text":" Stopping a code generator on triple backticks works until the model writes a nested code block. Stopping on "},{"text":"\"}\"","code":true},{"text":" breaks the first time the output contains nested JSON."}]},{"t":"h2","id":"the-security-consequence","runs":[{"text":"The security consequence"}]},{"t":"p","runs":[{"text":"Here is the part that matters beyond configuration. Because role markers are just tokens, text arriving from outside can contain them. If your application builds a prompt by pasting in a user’s message, a retrieved document, or a web page, and that text includes "},{"text":"<|im_start|>system","code":true},{"text":", you have handed a stranger the ability to open what looks like a system turn."}]},{"t":"p","runs":[{"text":"Most serving stacks defend against this by refusing to tokenise special-token strings from user content: the "},{"text":"allowed_special","code":true},{"text":" setting in some tokenizers exists exactly here. "},{"text":"Verify that your stack does this rather than assuming it.","b":true},{"text":" The check takes ten minutes: send a message containing your model’s role markers and inspect the token ids that reach the model."}]},{"t":"p","runs":[{"text":"This is one concrete instance of the general property that module six treats at length: there is no structural separation between instructions and data, only formatting conventions that the model learned to respect."}]},{"t":"h2","id":"the-five-minute-diagnostic","runs":[{"text":"The five-minute diagnostic"}]},{"t":"p","runs":[{"text":"When a local model behaves oddly, print the exact string that goes in:"}]},{"t":"code","lang":"python","label":"Python","text":"print(repr(tokenizer.apply_chat_template(messages, tokenize=False,\n                                         add_generation_prompt=True)))"},{"t":"p","runs":[{"text":"Compare it to the model card’s documented format, character by character. In my experience roughly half of “this open model is bad” reports are resolved at this step, and the other half by the quantisation issues in module seven."}]}],"check":{"question":[{"text":"A locally served model answers the user’s question and then writes a plausible follow-up question from the user, and answers that too. What is the cause?"}],"options":[{"text":[{"text":"The temperature is too high, so the model wanders past its intended answer."}],"correct":false,"tempting":[{"text":"High temperature does produce rambling, and turning it down is the reflexive first fix for strange output."}]},{"text":[{"text":"The context window includes previous turns that the model is continuing."}],"correct":false,"tempting":[{"text":"History leaking into a prompt is a real bug class, and it would explain conversational-looking output."}]},{"text":[{"text":"The maximum token limit is set too high, letting the model keep going."}],"correct":false,"tempting":[{"text":"The limit does bound length, so raising or lowering it feels like the obvious control over runaway output."}]},{"text":[{"text":"The end-of-turn token is not configured as a stop condition, so nothing halts generation when the assistant’s turn naturally ends."}],"correct":true,"tempting":null}],"order":[0,3,2,1],"why":null},"prev":{"slug":"reading-logprobs","title":"Reading the numbers the model will give you","minutes":9},"next":{"slug":"constrained-decoding","title":"Making output structurally valid, without asking nicely","minutes":9}}}