{"course":{"slug":"running-models-yourself","title":"Running Models Yourself","url":"https://addaly.com/learn/running-models-yourself","level":"intermediate","track":"deep"},"lesson":{"slug":"sampling-is-often-the-problem","title":"Sampling: why output quality is often not the model","url":"https://addaly.com/learn/running-models-yourself/sampling-is-often-the-problem","position":47,"of":82,"minutes":8,"answer":"The sampler chooses; before blaming the model, check the template, the context, and one knob at a time.","keyIdea":"The sampler chooses; before blaming the model, check the template, the context, and one knob at a time."},"body":{"format":"markdown","text":"A model does not produce text. It produces a probability distribution over the next token, about 32,000 to 150,000 numbers wide, and something else chooses. That chooser is the sampler, and a surprising share of \"this model is bad\" is a sampler set wrong.\n\n## The pipeline\n\nLogits come out of the model, then, in most backends: penalties are applied, temperature scales, truncation methods cut the tail, and one token is drawn from what remains. The order varies between backends and occasionally matters.\n\n> Figure: What happens between the model and the word you read. Steps: 1. Logits (One number per token, up to 150,000 of them); 2. Penalties (Repetition or DRY, over what is already written); 3. Temperature (Under one sharpens; zero is greedy); 4. Truncation (top-k, top-p or min-p cut the tail); 5. One token drawn (Then the whole thing runs again). The model produces a distribution; something else chooses. The order of these steps differs between backends, which is why the same nominal settings behave differently in two applications, and why a configuration copied from a forum post often does not reproduce.\n\n**temperature** divides the logits before softmax. Below 1 sharpens the distribution toward the top token; above 1 flattens it. At 0 it is greedy: always the most likely token, fully reproducible.\n\n**top-k** keeps the k highest-probability tokens. Blunt: it keeps 40 candidates whether the model is certain or lost.\n\n**top-p** (nucleus) keeps the smallest set of tokens whose probabilities sum to p. Adaptive, and the default almost everywhere. Its failure mode is the flat distribution: when the model is uncertain, p=0.95 can admit several hundred tokens, most of them bad.\n\n**min-p** keeps tokens whose probability is at least `min_p x p_max`. This scales with the model's own confidence (narrow when it is sure, wide when it is not) which is exactly the behaviour top-p fails at. `min_p 0.05` with `temperature 1.0` is a good modern default, and it tolerates high temperature far better.\n\n**repetition_penalty** divides the logits of tokens seen in the last N tokens. It is token-blind, so it also penalises `}`, `def`, `the`, and every newline. Above about 1.15 you can watch it break indentation and drop closing brackets.\n\n**DRY** (Don't Repeat Yourself) penalises only tokens that would *continue an already-repeated sequence*, with the penalty growing with match length. It stops loops without punishing the ordinary repetition that structured text requires. Usual settings: multiplier 0.8, base 1.75, allowed length 2.\n\n**XTC** (Exclude Top Choices) sometimes removes all but the least likely of the tokens above a probability threshold: deliberately throwing away the obvious continuation. It makes prose less predictable and makes anything that must be correct worse. Use it for fiction, never for code or extraction.\n\n## Settings by task\n\n**Extraction, classification, code, JSON:** temperature 0, or 0.2 with top_p 1.0. No repetition penalty. If your backend supports constrained decoding, use it: llama.cpp `--grammar`, vLLM guided decoding, SGLang's grammar backend. A schema is stronger than any prompt instruction.\n\n**General assistant:** temperature 0.7 with min_p 0.05, no repetition penalty. Or better, whatever the model card recommends: many now specify exact numbers, and those were tuned against the model's own training.\n\n**Creative writing:** temperature 1.0 to 1.2, min_p 0.05, DRY on, XTC optional.\n\n```bash\ncurl http://localhost:8080/v1/chat/completions -H 'Content-Type: application/json' -d '{\n  \"model\": \"local\",\n  \"messages\": [{\"role\": \"user\", \"content\": \"Return only JSON: {\\\"city\\\": ...}\"}],\n  \"temperature\": 0,\n  \"top_p\": 1.0\n}'\n```\n\n## Check these before touching the sampler\n\nMost bad output is not a sampler problem either. In rough order of frequency:\n\n1. **Wrong chat template.** The model was trained on specific role markers and is not seeing them. Output is subtly off in a way that looks like low intelligence.\n2. **Silent context truncation.** Lesson 4.\n3. **You loaded the base model, not the instruct model.** Base models continue text; they do not answer.\n4. **A system prompt you forgot about**, injected by your UI.\n\nWork down that list first. Then change one sampler value at a time, with a fixed seed and the same five prompts, and read the outputs. Changing four knobs at once and forming an impression is how people end up with settings they cannot explain and will not abandon."},"check":{"question":"Your model loops: \"I think that I think that I think...\". You raise repetition_penalty from 1.0 to 1.3. The loop stops, but generated code now has broken indentation and unclosed brackets. What happened?","options":[{"text":"Repetition penalty is token-blind, so it also suppressed the structural tokens code must repeat; DRY penalises repeated sequences instead and leaves those alone.","correct":true},{"text":"The penalty is still too weak, so the model is compensating in other ways; raising it further will settle the output.","correct":false,"tempting":"Reads a side effect of too much penalty as evidence of too little, and doubles the cause."},{"text":"Penalties are being applied after temperature scaling, so their effect is distorted; correcting the order fixes it.","correct":false,"tempting":"Sampler ordering is real and does matter in edge cases, but it does not produce this specific structural damage."},{"text":"1.3 is a normal value; the broken syntax means the 4-bit quantisation has damaged the model's grasp of code structure.","correct":false,"tempting":"Blames the weights for a change you just made to the sampler, which is the most common misdiagnosis in this whole area."}]},"prev":{"slug":"queueing-and-backpressure","title":"Queues, timeouts and refusing work gracefully","url":"https://addaly.com/learn/running-models-yourself/queueing-and-backpressure"},"next":{"slug":"advanced-samplers","title":"The samplers most interfaces do not show you","url":"https://addaly.com/learn/running-models-yourself/advanced-samplers"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Sampling: why output quality is often not the model', in Running Models Yourself. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/running-models-yourself/sampling-is-often-the-problem (Accessed: 31 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Sampling: why output quality is often not the model","courseTitle":"Running Models Yourself","module":{"index":6,"slug":"getting-good-output","title":"Getting good output"},"answer":null,"keyIdea":"The sampler chooses; before blaming the model, check the template, the context, and one knob at a time.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure 6.1","figure":{"type":"flow","steps":[{"label":"Logits","detail":"One number per token, up to 150,000 of them"},{"label":"Penalties","detail":"Repetition or DRY, over what is already written"},{"label":"Temperature","detail":"Under one sharpens; zero is greedy"},{"label":"Truncation","detail":"top-k, top-p or min-p cut the tail"},{"label":"One token drawn","detail":"Then the whole thing runs again"}],"title":"What happens between the model and the word you read","caption":"The model produces a distribution; something else chooses. The order of these steps differs between backends, which is why the same nominal settings behave differently in two applications, and why a configuration copied from a forum post often does not reproduce."},"alt":"Figure: What happens between the model and the word you read. Steps: 1. Logits (One number per token, up to 150,000 of them); 2. Penalties (Repetition or DRY, over what is already written); 3. Temperature (Under one sharpens; zero is greedy); 4. Truncation (top-k, top-p or min-p cut the tail); 5. One token drawn (Then the whole thing runs again). The model produces a distribution; something else chooses. The order of these steps differs between backends, which is why the same nominal settings behave differently in two applications, and why a configuration copied from a forum post often does not reproduce."}},"blocks":[{"t":"p","runs":[{"text":"A model does not produce text. It produces a probability distribution over the next token, about 32,000 to 150,000 numbers wide, and something else chooses. That chooser is the sampler, and a surprising share of “this model is bad” is a sampler set wrong."}],"lede":true},{"t":"h2","id":"the-pipeline","runs":[{"text":"The pipeline"}]},{"t":"p","runs":[{"text":"Logits come out of the model, then, in most backends: penalties are applied, temperature scales, truncation methods cut the tail, and one token is drawn from what remains. The order varies between backends and occasionally matters."}]},{"t":"p","runs":[{"text":"temperature","b":true},{"text":" divides the logits before softmax. Below 1 sharpens the distribution toward the top token; above 1 flattens it. At 0 it is greedy: always the most likely token, fully reproducible."}]},{"t":"p","runs":[{"text":"top-k","b":true},{"text":" keeps the k highest-probability tokens. Blunt: it keeps 40 candidates whether the model is certain or lost."}]},{"t":"p","runs":[{"text":"top-p","b":true},{"text":" (nucleus) keeps the smallest set of tokens whose probabilities sum to p. Adaptive, and the default almost everywhere. Its failure mode is the flat distribution: when the model is uncertain, p=0.95 can admit several hundred tokens, most of them bad."}]},{"t":"p","runs":[{"text":"min-p","b":true},{"text":" keeps tokens whose probability is at least "},{"text":"min_p x p_max","code":true},{"text":". This scales with the model’s own confidence (narrow when it is sure, wide when it is not) which is exactly the behaviour top-p fails at. "},{"text":"min_p 0.05","code":true},{"text":" with "},{"text":"temperature 1.0","code":true},{"text":" is a good modern default, and it tolerates high temperature far better."}]},{"t":"p","runs":[{"text":"repetition_penalty","b":true},{"text":" divides the logits of tokens seen in the last N tokens. It is token-blind, so it also penalises "},{"text":"}","code":true},{"text":", "},{"text":"def","code":true},{"text":", "},{"text":"the","code":true},{"text":", and every newline. Above about 1.15 you can watch it break indentation and drop closing brackets."}]},{"t":"p","runs":[{"text":"DRY","b":true},{"text":" (Don’t Repeat Yourself) penalises only tokens that would "},{"text":"continue an already-repeated sequence","i":true},{"text":", with the penalty growing with match length. It stops loops without punishing the ordinary repetition that structured text requires. Usual settings: multiplier 0.8, base 1.75, allowed length 2."}]},{"t":"p","runs":[{"text":"XTC","b":true},{"text":" (Exclude Top Choices) sometimes removes all but the least likely of the tokens above a probability threshold: deliberately throwing away the obvious continuation. It makes prose less predictable and makes anything that must be correct worse. Use it for fiction, never for code or extraction."}]},{"t":"h2","id":"settings-by-task","runs":[{"text":"Settings by task"}]},{"t":"p","runs":[{"text":"Extraction, classification, code, JSON:","b":true},{"text":" temperature 0, or 0.2 with top_p 1.0. No repetition penalty. If your backend supports constrained decoding, use it: llama.cpp "},{"text":"--grammar","code":true},{"text":", vLLM guided decoding, SGLang’s grammar backend. A schema is stronger than any prompt instruction."}]},{"t":"p","runs":[{"text":"General assistant:","b":true},{"text":" temperature 0.7 with min_p 0.05, no repetition penalty. Or better, whatever the model card recommends: many now specify exact numbers, and those were tuned against the model’s own training."}]},{"t":"p","runs":[{"text":"Creative writing:","b":true},{"text":" temperature 1.0 to 1.2, min_p 0.05, DRY on, XTC optional."}]},{"t":"code","lang":"bash","label":"Bash","text":"curl http://localhost:8080/v1/chat/completions -H 'Content-Type: application/json' -d '{\n  \"model\": \"local\",\n  \"messages\": [{\"role\": \"user\", \"content\": \"Return only JSON: {\\\"city\\\": ...}\"}],\n  \"temperature\": 0,\n  \"top_p\": 1.0\n}'"},{"t":"h2","id":"check-these-before-touching-the-sampler","runs":[{"text":"Check these before touching the sampler"}]},{"t":"p","runs":[{"text":"Most bad output is not a sampler problem either. In rough order of frequency:"}]},{"t":"ol","start":1,"items":[[{"text":"Wrong chat template.","b":true},{"text":" The model was trained on specific role markers and is not seeing them. Output is subtly off in a way that looks like low intelligence."}],[{"text":"Silent context truncation.","b":true},{"text":" Lesson 4."}],[{"text":"You loaded the base model, not the instruct model.","b":true},{"text":" Base models continue text; they do not answer."}],[{"text":"A system prompt you forgot about","b":true},{"text":", injected by your UI."}]]},{"t":"p","runs":[{"text":"Work down that list first. Then change one sampler value at a time, with a fixed seed and the same five prompts, and read the outputs. Changing four knobs at once and forming an impression is how people end up with settings they cannot explain and will not abandon."}]}],"check":{"question":[{"text":"Your model loops: “I think that I think that I think...”. You raise repetition_penalty from 1.0 to 1.3. The loop stops, but generated code now has broken indentation and unclosed brackets. What happened?"}],"options":[{"text":[{"text":"Repetition penalty is token-blind, so it also suppressed the structural tokens code must repeat; DRY penalises repeated sequences instead and leaves those alone."}],"correct":true,"tempting":null},{"text":[{"text":"The penalty is still too weak, so the model is compensating in other ways; raising it further will settle the output."}],"correct":false,"tempting":[{"text":"Reads a side effect of too much penalty as evidence of too little, and doubles the cause."}]},{"text":[{"text":"Penalties are being applied after temperature scaling, so their effect is distorted; correcting the order fixes it."}],"correct":false,"tempting":[{"text":"Sampler ordering is real and does matter in edge cases, but it does not produce this specific structural damage."}]},{"text":[{"text":"1.3 is a normal value; the broken syntax means the 4-bit quantisation has damaged the model’s grasp of code structure."}],"correct":false,"tempting":[{"text":"Blames the weights for a change you just made to the sampler, which is the most common misdiagnosis in this whole area."}]}],"order":[3,0,1,2],"why":null},"prev":{"slug":"queueing-and-backpressure","title":"Queues, timeouts and refusing work gracefully","minutes":8},"next":{"slug":"advanced-samplers","title":"The samplers most interfaces do not show you","minutes":9}}}