Why it will not stop talking

A specific, common, fixable failure

The model answers your question, and then keeps going: another paragraph, then a fabricated user message, then an answer to that, until the token limit ends it. Or it stops correctly in one tool and rambles in another with the same weights.

This is not the model being poor. It is a stop condition that is missing, wrong, or not being applied, and there are four places to look.

1. The end-of-turn token

Generation stops when the model emits a token the runtime recognises as terminal. Which token that is lives in the model’s configuration, and modern models often have more than one:

// generation_config.json, Llama 3.1
{
  "bos_token_id": 128000,
  "eos_token_id": [128001, 128008, 128009],
  "temperature": 0.6,
  "top_p": 0.9
}

Three end tokens: end-of-text, and two end-of-turn variants. If your runtime honours only the first, the model emits <|eot_id|> at the correct moment, the runtime does not recognise it, and generation continues. The text after that point is the model continuing a transcript, which is why it so often invents a new user turn.

The check is direct: log the raw token ids of a generation and see what it emitted where you wanted it to stop.

2. The template did not close the turn

If the prompt did not end with the assistant header (the add_generation_prompt problem from the template lesson) the model is not answering, it is continuing a document. Documents do not end after one reply.

3. Conversion or serving lost a token

A GGUF conversion that failed to carry across one of several end tokens produces exactly this symptom, and it is the most frequent defect in third-party conversions. Similarly, some serving layers apply generation_config.json and some ignore it entirely, using only eos_token_id from config.json.

Belt and braces, and worth doing on any model you did not convert yourself:

# llama.cpp: name the stop strings explicitly
llama-server -m model.gguf --chat-template llama3

# OpenAI-compatible clients: pass stop sequences
{"model": "...", "messages": [...], "stop": ["<|eot_id|>", "<|im_end|>"], "max_tokens": 512}

4. It is a base model

Base models have no notion of a turn ending. They continue until the limit. If you loaded -Base by accident, everything above is irrelevant and the fix is to load the instruct variant.

Stop strings, and their sharp edge

Most runtimes let you supply stop strings that halt generation when they appear in the output. They are matched on the decoded text, so they are useful for formats: stopping at a closing brace, at a section marker, at \nUser:.

Two cautions. The stop string is normally excluded from the output, so a stop at } may cut the brace you wanted. And a stop string that can appear inside legitimate output truncates good answers: stopping at a newline is a classic way to break every multi-line reply.

Always set a limit

Whatever else you do, set max_tokens. It is the backstop that turns a runaway into a truncated answer rather than a bill or a hung request. Reasoning models make this essential rather than prudent: a model that loops inside its thinking block can emit tens of thousands of tokens on a question it will never answer, and without a cap it will.

A reasonable default is twice the longest legitimate answer you expect. If you routinely hit the cap, that is data: either the task needs more room or something is looping.

The two-minute diagnosis

1. Does the output contain a special token as visible text, such as a literal <|eot_id|>? The runtime is not recognising it. Fix the stop configuration.
2. Does it write a new user turn? Check add_generation_prompt and the template.
3. Does it ramble without structure? Check whether you loaded a base model.
4. Does it stop correctly in transformers and not in your server? The serving layer is ignoring generation_config.json. Pass stops explicitly.

Four checks, each about thirty seconds, and one of them is nearly always the answer.