{"course":{"slug":"running-models-yourself","title":"Running Models Yourself","url":"https://addaly.com/learn/running-models-yourself","level":"intermediate","track":"deep"},"lesson":{"slug":"advanced-samplers","title":"The samplers most interfaces do not show you","url":"https://addaly.com/learn/running-models-yourself/advanced-samplers","position":48,"of":82,"minutes":9,"answer":"DRY penalises repeated sequences rather than repeated tokens, which is why it suppresses looping without breaking brackets and indentation the way repetition_penalty does at any effective setting.","keyIdea":"DRY penalises repeated sequences rather than repeated tokens, which is why it suppresses looping without breaking brackets and indentation the way repetition_penalty does at any effective setting."},"body":{"format":"markdown","text":"## The repetition problem, stated properly\n\nA model that has just written a phrase finds that phrase more likely again, because it now appears in its own context. Left alone, this compounds: the model repeats a sentence, then a paragraph, then loops until it hits the token limit. Small models and quantised models do it more.\n\nThe classic remedy, `repetition_penalty`, divides the logits of tokens seen in the last N tokens. It works and it is blunt in a specific way worth understanding: it is token-blind. It penalises `}`, `def`, `the`, `import`, and every newline exactly as hard as it penalises the phrase you wanted to suppress. Above about 1.15 you can watch it break indentation, drop closing brackets and produce increasingly odd word choices, because the tokens that must repeat are being punished for repeating.\n\nTwo newer samplers fix this properly, and neither appears in most graphical interfaces.\n\n## DRY: penalise repeated sequences, not repeated tokens\n\nDRY (\"Do not Repeat Yourself\") looks for the longest suffix of the generated text that has occurred earlier, and penalises only the token that would extend that repetition. The penalty grows exponentially with the length of the repeated sequence.\n\nThe difference is decisive. A single `}` that has appeared two hundred times is untouched, because `}` alone is not a repeating sequence in context. A four-word phrase beginning to repeat for the third time is heavily penalised. You get repetition control without syntax damage, which `repetition_penalty` cannot offer at any setting.\n\n```json\n{\"dry_multiplier\": 0.8, \"dry_base\": 1.75, \"dry_allowed_length\": 2}\n```\n\n`dry_allowed_length` is how long a repeat may be before penalty starts. Two is a good default; raise it for structured output where longer legitimate repeats occur.\n\n## XTC: remove the top choice sometimes\n\nXTC (\"Exclude Top Choices\") inverts the usual approach. Instead of truncating the tail, it occasionally removes the *most* likely tokens when several are already above a probability threshold, forcing a less predictable choice.\n\nThe reasoning: when a model is confident about several options, the top one is usually the blandest, most generic continuation. Removing it produces noticeably more varied text.\n\nThis is for creative writing and only for creative writing. On code, extraction or factual answers, deliberately discarding the model's best token is precisely wrong. Two parameters, and both matter:\n\n```json\n{\"xtc_threshold\": 0.1, \"xtc_probability\": 0.5}\n```\n\nOnly tokens above the threshold are candidates for removal, and the removal happens with the given probability, so the top token still wins half the time.\n\n## Typical-p and Mirostat\n\n**Typical-p** keeps tokens whose information content is close to the distribution's expected information content: discarding both the boringly predictable and the wildly improbable. It produces text with steadier surprise, and it is worth trying when top-p output alternates between flat and unhinged.\n\n**Mirostat** takes a different approach entirely: rather than fixing a truncation rule, it targets a perplexity level and adjusts as it generates, using feedback to hold the output at a chosen level of surprise. It largely replaces top-k and top-p when active. Some people find it excellent for long-form prose; it interacts confusingly with other samplers, so use it alone.\n\n## Order matters, and is not standard\n\nThese are applied in a sequence, and the sequence differs between backends. Penalties before or after temperature changes their strength. Truncation before or after temperature changes what survives. llama.cpp exposes `--samplers` to set the order explicitly:\n\n```bash\n--samplers \"dry;top_k;typ_p;top_p;min_p;xtc;temperature\"\n```\n\nThis is why the same nominal settings can behave differently in two applications, and it is the answer when a configuration copied from a forum post does not reproduce.\n\n## Sane starting points\n\n- **Factual, extraction, classification, code**: `temperature 0`. Nothing else matters, because there is no sampling to control. Reproducible, and it is the right default for anything a program consumes.\n- **General assistant**: `temperature 0.7`, `min_p 0.05`, everything else off. Add `dry_multiplier 0.8` if it loops.\n- **Creative prose**: `temperature 1.0`, `min_p 0.05`, `dry_multiplier 0.8`, and `xtc_threshold 0.1` with `xtc_probability 0.5` if the writing feels generic.\n\nChange one at a time, with the same prompt, and read the output. Sampler tuning is the part of this subject where people spend the most effort for the least measured benefit: get to a sane default, verify the template is right, and move on."},"check":{"question":"A code-generation setup loops on boilerplate. Raising repetition_penalty to 1.25 stops the looping but the generated Python now has inconsistent indentation and occasional missing closing brackets. What explains this and what is the better tool?","options":[{"text":"The penalty window is too short, so it wraps around and penalises tokens from the current line; lengthening the window fixes it.","correct":false,"tempting":"Invents a wrapping behaviour; a longer window penalises more tokens, which would worsen the syntax damage rather than fix it."},{"text":"The penalty applies per token regardless of context, so structurally necessary repeats like whitespace and brackets are suppressed too; DRY penalises repeated sequences instead and leaves single recurring tokens alone.","correct":true},{"text":"High repetition penalty interacts badly with top-p, and switching to min_p at the same penalty will restore valid syntax.","correct":false,"tempting":"Changes the truncation method while leaving the token-blind penalty that is causing the damage entirely in place."},{"text":"Code models require temperature 0, and any penalty at all is inappropriate for structured output.","correct":false,"tempting":"Overcorrects into a rule that would leave the original looping problem unaddressed at temperature 0, where it can still occur."}]},"prev":{"slug":"sampling-is-often-the-problem","title":"Sampling: why output quality is often not the model","url":"https://addaly.com/learn/running-models-yourself/sampling-is-often-the-problem"},"next":{"slug":"prompting-a-small-model","title":"Prompting a small model is a different craft","url":"https://addaly.com/learn/running-models-yourself/prompting-a-small-model"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'The samplers most interfaces do not show you', in Running Models Yourself. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/running-models-yourself/advanced-samplers (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"The samplers most interfaces do not show you","courseTitle":"Running Models Yourself","module":{"index":6,"slug":"getting-good-output","title":"Getting good output"},"answer":null,"keyIdea":"DRY penalises repeated sequences rather than repeated tokens, which is why it suppresses looping without breaking brackets and indentation the way repetition_penalty does at any effective setting.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"the-repetition-problem-stated-properly","runs":[{"text":"The repetition problem, stated properly"}]},{"t":"p","runs":[{"text":"A model that has just written a phrase finds that phrase more likely again, because it now appears in its own context. Left alone, this compounds: the model repeats a sentence, then a paragraph, then loops until it hits the token limit. Small models and quantised models do it more."}]},{"t":"p","runs":[{"text":"The classic remedy, "},{"text":"repetition_penalty","code":true},{"text":", divides the logits of tokens seen in the last N tokens. It works and it is blunt in a specific way worth understanding: it is token-blind. It penalises "},{"text":"}","code":true},{"text":", "},{"text":"def","code":true},{"text":", "},{"text":"the","code":true},{"text":", "},{"text":"import","code":true},{"text":", and every newline exactly as hard as it penalises the phrase you wanted to suppress. Above about 1.15 you can watch it break indentation, drop closing brackets and produce increasingly odd word choices, because the tokens that must repeat are being punished for repeating."}]},{"t":"p","runs":[{"text":"Two newer samplers fix this properly, and neither appears in most graphical interfaces."}]},{"t":"h2","id":"dry-penalise-repeated-sequences-not-repeated-tokens","runs":[{"text":"DRY: penalise repeated sequences, not repeated tokens"}]},{"t":"p","runs":[{"text":"DRY (“Do not Repeat Yourself”) looks for the longest suffix of the generated text that has occurred earlier, and penalises only the token that would extend that repetition. The penalty grows exponentially with the length of the repeated sequence."}]},{"t":"p","runs":[{"text":"The difference is decisive. A single "},{"text":"}","code":true},{"text":" that has appeared two hundred times is untouched, because "},{"text":"}","code":true},{"text":" alone is not a repeating sequence in context. A four-word phrase beginning to repeat for the third time is heavily penalised. You get repetition control without syntax damage, which "},{"text":"repetition_penalty","code":true},{"text":" cannot offer at any setting."}]},{"t":"code","lang":"json","label":"JSON","text":"{\"dry_multiplier\": 0.8, \"dry_base\": 1.75, \"dry_allowed_length\": 2}"},{"t":"p","runs":[{"text":"dry_allowed_length","code":true},{"text":" is how long a repeat may be before penalty starts. Two is a good default; raise it for structured output where longer legitimate repeats occur."}]},{"t":"h2","id":"xtc-remove-the-top-choice-sometimes","runs":[{"text":"XTC: remove the top choice sometimes"}]},{"t":"p","runs":[{"text":"XTC (“Exclude Top Choices”) inverts the usual approach. Instead of truncating the tail, it occasionally removes the "},{"text":"most","i":true},{"text":" likely tokens when several are already above a probability threshold, forcing a less predictable choice."}]},{"t":"p","runs":[{"text":"The reasoning: when a model is confident about several options, the top one is usually the blandest, most generic continuation. Removing it produces noticeably more varied text."}]},{"t":"p","runs":[{"text":"This is for creative writing and only for creative writing. On code, extraction or factual answers, deliberately discarding the model’s best token is precisely wrong. Two parameters, and both matter:"}]},{"t":"code","lang":"json","label":"JSON","text":"{\"xtc_threshold\": 0.1, \"xtc_probability\": 0.5}"},{"t":"p","runs":[{"text":"Only tokens above the threshold are candidates for removal, and the removal happens with the given probability, so the top token still wins half the time."}]},{"t":"h2","id":"typical-p-and-mirostat","runs":[{"text":"Typical-p and Mirostat"}]},{"t":"p","runs":[{"text":"Typical-p","b":true},{"text":" keeps tokens whose information content is close to the distribution’s expected information content: discarding both the boringly predictable and the wildly improbable. It produces text with steadier surprise, and it is worth trying when top-p output alternates between flat and unhinged."}]},{"t":"p","runs":[{"text":"Mirostat","b":true},{"text":" takes a different approach entirely: rather than fixing a truncation rule, it targets a perplexity level and adjusts as it generates, using feedback to hold the output at a chosen level of surprise. It largely replaces top-k and top-p when active. Some people find it excellent for long-form prose; it interacts confusingly with other samplers, so use it alone."}]},{"t":"h2","id":"order-matters-and-is-not-standard","runs":[{"text":"Order matters, and is not standard"}]},{"t":"p","runs":[{"text":"These are applied in a sequence, and the sequence differs between backends. Penalties before or after temperature changes their strength. Truncation before or after temperature changes what survives. llama.cpp exposes "},{"text":"--samplers","code":true},{"text":" to set the order explicitly:"}]},{"t":"code","lang":"bash","label":"Bash","text":"--samplers \"dry;top_k;typ_p;top_p;min_p;xtc;temperature\""},{"t":"p","runs":[{"text":"This is why the same nominal settings can behave differently in two applications, and it is the answer when a configuration copied from a forum post does not reproduce."}]},{"t":"h2","id":"sane-starting-points","runs":[{"text":"Sane starting points"}]},{"t":"ul","items":[[{"text":"Factual, extraction, classification, code","b":true},{"text":": "},{"text":"temperature 0","code":true},{"text":". Nothing else matters, because there is no sampling to control. Reproducible, and it is the right default for anything a program consumes."}],[{"text":"General assistant","b":true},{"text":": "},{"text":"temperature 0.7","code":true},{"text":", "},{"text":"min_p 0.05","code":true},{"text":", everything else off. Add "},{"text":"dry_multiplier 0.8","code":true},{"text":" if it loops."}],[{"text":"Creative prose","b":true},{"text":": "},{"text":"temperature 1.0","code":true},{"text":", "},{"text":"min_p 0.05","code":true},{"text":", "},{"text":"dry_multiplier 0.8","code":true},{"text":", and "},{"text":"xtc_threshold 0.1","code":true},{"text":" with "},{"text":"xtc_probability 0.5","code":true},{"text":" if the writing feels generic."}]]},{"t":"p","runs":[{"text":"Change one at a time, with the same prompt, and read the output. Sampler tuning is the part of this subject where people spend the most effort for the least measured benefit: get to a sane default, verify the template is right, and move on."}]}],"check":{"question":[{"text":"A code-generation setup loops on boilerplate. Raising repetition_penalty to 1.25 stops the looping but the generated Python now has inconsistent indentation and occasional missing closing brackets. What explains this and what is the better tool?"}],"options":[{"text":[{"text":"The penalty window is too short, so it wraps around and penalises tokens from the current line; lengthening the window fixes it."}],"correct":false,"tempting":[{"text":"Invents a wrapping behaviour; a longer window penalises more tokens, which would worsen the syntax damage rather than fix it."}]},{"text":[{"text":"The penalty applies per token regardless of context, so structurally necessary repeats like whitespace and brackets are suppressed too; DRY penalises repeated sequences instead and leaves single recurring tokens alone."}],"correct":true,"tempting":null},{"text":[{"text":"High repetition penalty interacts badly with top-p, and switching to min_p at the same penalty will restore valid syntax."}],"correct":false,"tempting":[{"text":"Changes the truncation method while leaving the token-blind penalty that is causing the damage entirely in place."}]},{"text":[{"text":"Code models require temperature 0, and any penalty at all is inappropriate for structured output."}],"correct":false,"tempting":[{"text":"Overcorrects into a rule that would leave the original looping problem unaddressed at temperature 0, where it can still occur."}]}],"order":[2,3,0,1],"why":null},"prev":{"slug":"sampling-is-often-the-problem","title":"Sampling: why output quality is often not the model","minutes":8},"next":{"slug":"prompting-a-small-model","title":"Prompting a small model is a different craft","minutes":9}}}