Special tokens, stop conditions, and where a turn ends

Nothing tells the model the turn is over

Generation runs until something stops it. There are exactly three stopping conditions, and they are all mundane:

1. The model emits its end-of-turn token, and the serving code recognises it and halts.
2. The generated text matches a stop sequence you supplied.
3. The maximum token count is reached.

That is all. There is no notion of a complete thought. A response that ends mid-sentence hit condition 3, and the fix is a larger limit, not a better prompt.

Special tokens are ordinary integers

The markers around roles (written variously as <|im_start|>, <|eot_id|>, <|end|>) are entries in the vocabulary like any other. They have ids, embeddings, and logits. What makes them special is only that training data was formatted with them, so the model learned to emit an end-of-turn token where an assistant reply naturally finishes.

Check generation_config.json in any open model and you will find eos_token_id. Some models have several valid stop tokens, and a serving stack configured with only one will produce output that runs on and on, often with the model helpfully inventing the user’s next message. That symptom is diagnostic: if your model writes both sides of the conversation, your stop token configuration is wrong.

Why the template must be exact

Module one showed the chat template lives in tokenizer_config.json. Two failure modes follow from getting it wrong, and both are common:

Wrong markers. Use one family’s template with another family’s model and the model sees a formatting it never trained on. It still produces text, but the assistant behaviour it learned is keyed to those markers, so quality degrades in a way that looks like the model being poor rather than misconfigured.

Missing generation prompt. After the last user message, the template must add the opening marker for the assistant turn. Omit it and the model is being asked to continue the user’s message, so it writes another user question. In Hugging Face this is add_generation_prompt=True, and forgetting it is a rite of passage.

prompt = tokenizer.apply_chat_template(
    messages, tokenize=False, add_generation_prompt=True
)

Stop sequences, and the trap in them

Stop sequences are matched on the decoded text, not on tokens, so they can trigger mid-token in ways that look strange. Two rules:

The stop sequence is not returned. If you stop on "\n\n" you get the text up to it, and you will need to add it back if your format expects it.

Beware stopping on something that can legitimately appear. Stopping a code generator on triple backticks works until the model writes a nested code block. Stopping on "}" breaks the first time the output contains nested JSON.

The security consequence

Here is the part that matters beyond configuration. Because role markers are just tokens, text arriving from outside can contain them. If your application builds a prompt by pasting in a user’s message, a retrieved document, or a web page, and that text includes <|im_start|>system, you have handed a stranger the ability to open what looks like a system turn.

Most serving stacks defend against this by refusing to tokenise special-token strings from user content: the allowed_special setting in some tokenizers exists exactly here. Verify that your stack does this rather than assuming it. The check takes ten minutes: send a message containing your model’s role markers and inspect the token ids that reach the model.

This is one concrete instance of the general property that module six treats at length: there is no structural separation between instructions and data, only formatting conventions that the model learned to respect.

The five-minute diagnostic

When a local model behaves oddly, print the exact string that goes in:

print(repr(tokenizer.apply_chat_template(messages, tokenize=False,
                                         add_generation_prompt=True)))

Compare it to the model card’s documented format, character by character. In my experience roughly half of “this open model is bad” reports are resolved at this step, and the other half by the quantisation issues in module seven.