{"course":{"slug":"open-models","title":"The Open Model Ecosystem","url":"https://addaly.com/learn/open-models","level":"intermediate","track":"deep"},"lesson":{"slug":"stop-conditions","title":"Why it will not stop talking","url":"https://addaly.com/learn/open-models/stop-conditions","position":30,"of":71,"minutes":8,"answer":"Runaway generation is almost always a stop condition rather than a bad model: several end-of-turn tokens, a template that never opened the assistant turn, a lossy conversion, or a serving layer ignoring generation_config.json.","keyIdea":"Runaway generation is almost always a stop condition rather than a bad model: several end-of-turn tokens, a template that never opened the assistant turn, a lossy conversion, or a serving layer ignoring generation_config.json."},"body":{"format":"markdown","text":"## A specific, common, fixable failure\n\nThe model answers your question, and then keeps going: another paragraph, then a fabricated user message, then an answer to that, until the token limit ends it. Or it stops correctly in one tool and rambles in another with the same weights.\n\nThis is not the model being poor. It is a stop condition that is missing, wrong, or not being applied, and there are four places to look.\n\n> Figure: Four checks, about thirty seconds each. Steps: 1. Is a special token visible as text? (A literal end-of-turn marker in the output means the runtime does not recognise it. Fix the stop configuration); 2. Did it write a new user turn? (The prompt did not end with the assistant header. Check add_generation_prompt and the template); 3. Does it ramble with no structure? (You loaded the base model, which has no notion of a turn ending. Load the instruct variant); 4. Right in transformers, wrong in your server? (The serving layer is ignoring generation_config.json. Pass the stop sequences explicitly). One of them is nearly always the answer. Whatever the outcome, set max_tokens as well: it turns a runaway into a truncated answer rather than a bill, and a reasoning model looping inside its thinking block will emit tens of thousands of tokens without one.\n\n## 1. The end-of-turn token\n\nGeneration stops when the model emits a token the runtime recognises as terminal. Which token that is lives in the model's configuration, and modern models often have more than one:\n\n```json\n// generation_config.json, Llama 3.1\n{\n  \"bos_token_id\": 128000,\n  \"eos_token_id\": [128001, 128008, 128009],\n  \"temperature\": 0.6,\n  \"top_p\": 0.9\n}\n```\n\nThree end tokens: end-of-text, and two end-of-turn variants. If your runtime honours only the first, the model emits `<|eot_id|>` at the correct moment, the runtime does not recognise it, and generation continues. The text after that point is the model continuing a transcript, which is why it so often invents a new user turn.\n\nThe check is direct: log the raw token ids of a generation and see what it emitted where you wanted it to stop.\n\n## 2. The template did not close the turn\n\nIf the prompt did not end with the assistant header (the `add_generation_prompt` problem from the template lesson) the model is not answering, it is continuing a document. Documents do not end after one reply.\n\n## 3. Conversion or serving lost a token\n\nA GGUF conversion that failed to carry across one of several end tokens produces exactly this symptom, and it is the most frequent defect in third-party conversions. Similarly, some serving layers apply `generation_config.json` and some ignore it entirely, using only `eos_token_id` from `config.json`.\n\nBelt and braces, and worth doing on any model you did not convert yourself:\n\n```bash\n# llama.cpp: name the stop strings explicitly\nllama-server -m model.gguf --chat-template llama3\n\n# OpenAI-compatible clients: pass stop sequences\n{\"model\": \"...\", \"messages\": [...], \"stop\": [\"<|eot_id|>\", \"<|im_end|>\"], \"max_tokens\": 512}\n```\n\n## 4. It is a base model\n\nBase models have no notion of a turn ending. They continue until the limit. If you loaded `-Base` by accident, everything above is irrelevant and the fix is to load the instruct variant.\n\n## Stop strings, and their sharp edge\n\nMost runtimes let you supply stop strings that halt generation when they appear in the output. They are matched on the decoded text, so they are useful for formats: stopping at a closing brace, at a section marker, at `\\nUser:`.\n\nTwo cautions. The stop string is normally excluded from the output, so a stop at `}` may cut the brace you wanted. And a stop string that can appear inside legitimate output truncates good answers: stopping at a newline is a classic way to break every multi-line reply.\n\n## Always set a limit\n\nWhatever else you do, set `max_tokens`. It is the backstop that turns a runaway into a truncated answer rather than a bill or a hung request. Reasoning models make this essential rather than prudent: a model that loops inside its thinking block can emit tens of thousands of tokens on a question it will never answer, and without a cap it will.\n\nA reasonable default is twice the longest legitimate answer you expect. If you routinely hit the cap, that is data: either the task needs more room or something is looping.\n\n## The two-minute diagnosis\n\n1. Does the output contain a special token as visible text, such as a literal `<|eot_id|>`? The runtime is not recognising it. Fix the stop configuration.\n2. Does it write a new user turn? Check `add_generation_prompt` and the template.\n3. Does it ramble without structure? Check whether you loaded a base model.\n4. Does it stop correctly in `transformers` and not in your server? The serving layer is ignoring `generation_config.json`. Pass stops explicitly.\n\nFour checks, each about thirty seconds, and one of them is nearly always the answer."},"check":{"question":"A model stops correctly when run through transformers but rambles when served through a local OpenAI-compatible server, with a literal <|eot_id|> appearing in the output text. What does that tell you?","options":[{"text":"The server is generating past the model's context window, so tokens are being decoded from corrupted state.","correct":false,"tempting":"Context overflow produces degraded or truncated text, not a correctly emitted special token rendered as visible characters."},{"text":"The model file is corrupt, since a valid model would never emit a special token as text.","correct":false,"tempting":"The model emitted the right token at the right moment; the fault is entirely on the decoding and stop side."},{"text":"The server is not treating that token as terminal, so the model's correct stop signal is being decoded as ordinary text and generation continues.","correct":true},{"text":"The chat template is wrong, since a correct template would prevent the token appearing in the output.","correct":false,"tempting":"A template fault usually shows as the model writing a new user turn; here the model signalled the end properly and the server ignored it."}]},"prev":{"slug":"weight-formats-and-conversion","title":"Formats, and what conversion costs you","url":"https://addaly.com/learn/open-models/weight-formats-and-conversion"},"next":{"slug":"choosing-for-the-job","title":"Choosing a Model for the Job","url":"https://addaly.com/learn/open-models/choosing-for-the-job"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Why it will not stop talking', in The Open Model Ecosystem. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/open-models/stop-conditions (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Why it will not stop talking","courseTitle":"The Open Model Ecosystem","module":{"index":3,"slug":"inside-the-artefact","title":"Inside the files"},"answer":null,"keyIdea":"Runaway generation is almost always a stop condition rather than a bad model: several end-of-turn tokens, a template that never opened the assistant turn, a lossy conversion, or a serving layer ignoring generation_config.json.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"type":"flow","steps":[{"label":"Is a special token visible as text?","detail":"A literal end-of-turn marker in the output means the runtime does not recognise it. Fix the stop configuration"},{"label":"Did it write a new user turn?","detail":"The prompt did not end with the assistant header. Check add_generation_prompt and the template"},{"label":"Does it ramble with no structure?","detail":"You loaded the base model, which has no notion of a turn ending. Load the instruct variant"},{"label":"Right in transformers, wrong in your server?","detail":"The serving layer is ignoring generation_config.json. Pass the stop sequences explicitly"}],"title":"Four checks, about thirty seconds each","caption":"One of them is nearly always the answer. Whatever the outcome, set max_tokens as well: it turns a runaway into a truncated answer rather than a bill, and a reasoning model looping inside its thinking block will emit tens of thousands of tokens without one."},"alt":"Figure: Four checks, about thirty seconds each. Steps: 1. Is a special token visible as text? (A literal end-of-turn marker in the output means the runtime does not recognise it. Fix the stop configuration); 2. Did it write a new user turn? (The prompt did not end with the assistant header. Check add_generation_prompt and the template); 3. Does it ramble with no structure? (You loaded the base model, which has no notion of a turn ending. Load the instruct variant); 4. Right in transformers, wrong in your server? (The serving layer is ignoring generation_config.json. Pass the stop sequences explicitly). One of them is nearly always the answer. Whatever the outcome, set max_tokens as well: it turns a runaway into a truncated answer rather than a bill, and a reasoning model looping inside its thinking block will emit tens of thousands of tokens without one."}},"blocks":[{"t":"h2","id":"a-specific-common-fixable-failure","runs":[{"text":"A specific, common, fixable failure"}]},{"t":"p","runs":[{"text":"The model answers your question, and then keeps going: another paragraph, then a fabricated user message, then an answer to that, until the token limit ends it. Or it stops correctly in one tool and rambles in another with the same weights."}]},{"t":"p","runs":[{"text":"This is not the model being poor. It is a stop condition that is missing, wrong, or not being applied, and there are four places to look."}]},{"t":"h2","id":"1-the-end-of-turn-token","runs":[{"text":"1. The end-of-turn token"}]},{"t":"p","runs":[{"text":"Generation stops when the model emits a token the runtime recognises as terminal. Which token that is lives in the model’s configuration, and modern models often have more than one:"}]},{"t":"code","lang":"json","label":"JSON","text":"// generation_config.json, Llama 3.1\n{\n  \"bos_token_id\": 128000,\n  \"eos_token_id\": [128001, 128008, 128009],\n  \"temperature\": 0.6,\n  \"top_p\": 0.9\n}"},{"t":"p","runs":[{"text":"Three end tokens: end-of-text, and two end-of-turn variants. If your runtime honours only the first, the model emits "},{"text":"<|eot_id|>","code":true},{"text":" at the correct moment, the runtime does not recognise it, and generation continues. The text after that point is the model continuing a transcript, which is why it so often invents a new user turn."}]},{"t":"p","runs":[{"text":"The check is direct: log the raw token ids of a generation and see what it emitted where you wanted it to stop."}]},{"t":"h2","id":"2-the-template-did-not-close-the-turn","runs":[{"text":"2. The template did not close the turn"}]},{"t":"p","runs":[{"text":"If the prompt did not end with the assistant header (the "},{"text":"add_generation_prompt","code":true},{"text":" problem from the template lesson) the model is not answering, it is continuing a document. Documents do not end after one reply."}]},{"t":"h2","id":"3-conversion-or-serving-lost-a-token","runs":[{"text":"3. Conversion or serving lost a token"}]},{"t":"p","runs":[{"text":"A GGUF conversion that failed to carry across one of several end tokens produces exactly this symptom, and it is the most frequent defect in third-party conversions. Similarly, some serving layers apply "},{"text":"generation_config.json","code":true},{"text":" and some ignore it entirely, using only "},{"text":"eos_token_id","code":true},{"text":" from "},{"text":"config.json","code":true},{"text":"."}]},{"t":"p","runs":[{"text":"Belt and braces, and worth doing on any model you did not convert yourself:"}]},{"t":"code","lang":"bash","label":"Bash","text":"# llama.cpp: name the stop strings explicitly\nllama-server -m model.gguf --chat-template llama3\n\n# OpenAI-compatible clients: pass stop sequences\n{\"model\": \"...\", \"messages\": [...], \"stop\": [\"<|eot_id|>\", \"<|im_end|>\"], \"max_tokens\": 512}"},{"t":"h2","id":"4-it-is-a-base-model","runs":[{"text":"4. It is a base model"}]},{"t":"p","runs":[{"text":"Base models have no notion of a turn ending. They continue until the limit. If you loaded "},{"text":"-Base","code":true},{"text":" by accident, everything above is irrelevant and the fix is to load the instruct variant."}]},{"t":"h2","id":"stop-strings-and-their-sharp-edge","runs":[{"text":"Stop strings, and their sharp edge"}]},{"t":"p","runs":[{"text":"Most runtimes let you supply stop strings that halt generation when they appear in the output. They are matched on the decoded text, so they are useful for formats: stopping at a closing brace, at a section marker, at "},{"text":"\\nUser:","code":true},{"text":"."}]},{"t":"p","runs":[{"text":"Two cautions. The stop string is normally excluded from the output, so a stop at "},{"text":"}","code":true},{"text":" may cut the brace you wanted. And a stop string that can appear inside legitimate output truncates good answers: stopping at a newline is a classic way to break every multi-line reply."}]},{"t":"h2","id":"always-set-a-limit","runs":[{"text":"Always set a limit"}]},{"t":"p","runs":[{"text":"Whatever else you do, set "},{"text":"max_tokens","code":true},{"text":". It is the backstop that turns a runaway into a truncated answer rather than a bill or a hung request. Reasoning models make this essential rather than prudent: a model that loops inside its thinking block can emit tens of thousands of tokens on a question it will never answer, and without a cap it will."}]},{"t":"p","runs":[{"text":"A reasonable default is twice the longest legitimate answer you expect. If you routinely hit the cap, that is data: either the task needs more room or something is looping."}]},{"t":"h2","id":"the-two-minute-diagnosis","runs":[{"text":"The two-minute diagnosis"}]},{"t":"ol","start":1,"items":[[{"text":"Does the output contain a special token as visible text, such as a literal "},{"text":"<|eot_id|>","code":true},{"text":"? The runtime is not recognising it. Fix the stop configuration."}],[{"text":"Does it write a new user turn? Check "},{"text":"add_generation_prompt","code":true},{"text":" and the template."}],[{"text":"Does it ramble without structure? Check whether you loaded a base model."}],[{"text":"Does it stop correctly in "},{"text":"transformers","code":true},{"text":" and not in your server? The serving layer is ignoring "},{"text":"generation_config.json","code":true},{"text":". Pass stops explicitly."}]]},{"t":"p","runs":[{"text":"Four checks, each about thirty seconds, and one of them is nearly always the answer."}]}],"check":{"question":[{"text":"A model stops correctly when run through transformers but rambles when served through a local OpenAI-compatible server, with a literal <|eot_id|> appearing in the output text. What does that tell you?"}],"options":[{"text":[{"text":"The server is generating past the model’s context window, so tokens are being decoded from corrupted state."}],"correct":false,"tempting":[{"text":"Context overflow produces degraded or truncated text, not a correctly emitted special token rendered as visible characters."}]},{"text":[{"text":"The model file is corrupt, since a valid model would never emit a special token as text."}],"correct":false,"tempting":[{"text":"The model emitted the right token at the right moment; the fault is entirely on the decoding and stop side."}]},{"text":[{"text":"The server is not treating that token as terminal, so the model’s correct stop signal is being decoded as ordinary text and generation continues."}],"correct":true,"tempting":null},{"text":[{"text":"The chat template is wrong, since a correct template would prevent the token appearing in the output."}],"correct":false,"tempting":[{"text":"A template fault usually shows as the model writing a new user turn; here the model signalled the end properly and the server ignored it."}]}],"order":[0,2,3,1],"why":null},"prev":{"slug":"weight-formats-and-conversion","title":"Formats, and what conversion costs you","minutes":8},"next":{"slug":"choosing-for-the-job","title":"Choosing a Model for the Job","minutes":9}}}