{"course":{"slug":"open-models","title":"The Open Model Ecosystem","url":"https://addaly.com/learn/open-models","level":"intermediate","track":"deep"},"lesson":{"slug":"sampling-parameters","title":"The sampler, and the defaults that ship broken","url":"https://addaly.com/learn/open-models/sampling-parameters","position":41,"of":71,"minutes":9,"answer":"The sampler converts logits into a token, and the common default of a repetition penalty above 1 actively damages JSON and code because those formats require repeated punctuation tokens.","keyIdea":"The sampler converts logits into a token, and the common default of a repetition penalty above 1 actively damages JSON and code because those formats require repeated punctuation tokens."},"body":{"format":"markdown","text":"## What happens after the forward pass\n\nThe model outputs a logit for every token in the vocabulary: 150,000 raw scores. The sampler turns those into one chosen token, and every parameter you have heard of is a step in that conversion.\n\n**Temperature** divides the logits before the softmax. Below 1 the distribution sharpens and the likeliest tokens dominate; above 1 it flattens and unlikely tokens get a chance. At 0 most implementations skip sampling entirely and take the highest-scoring token, which is greedy decoding.\n\n**top_k** keeps only the k highest-scoring tokens and renormalises. Blunt: k=40 is far too many when the model is certain and far too few when it is uncertain across a wide field.\n\n**top_p**, or nucleus sampling, keeps the smallest set of tokens whose probabilities sum to p. Adaptive in the right direction: a confident step keeps two tokens, an open one keeps forty.\n\n**min_p** keeps tokens whose probability is at least some fraction of the top token's probability. At min_p 0.05, if the best token has probability 0.6, nothing below 0.03 survives. It behaves better than top_p at high temperature, which is why creative-writing setups have converged on it.\n\n**repetition_penalty** divides the logits of tokens that have already appeared, with values above 1 discouraging them. **frequency_penalty** and **presence_penalty** do a similar job additively.\n\n## The default that quietly ruins structured output\n\nMost local interfaces ship with something like temperature 0.8, top_p 0.95 and a repetition penalty around 1.1. That is a reasonable configuration for chat and a bad one for everything else, and the repetition penalty is the actively harmful part.\n\nConsider what it does to JSON. The tokens `\"`, `:`, `,`, `{` and `}` must repeat: that is what the format is. A repetition penalty pushes their logits down every time they appear, so by the fifth field the model is being actively discouraged from producing correct syntax. The same applies to code, where indentation and brackets repeat by necessity, and to any tabular or list output.\n\n**Set repetition penalty to 1.0 for structured output.** If the model loops, the cause is usually the prompt or a missing stop token, and the penalty is treating the symptom by damaging the format.\n\n> Figure: Two sets of sampler settings, and why the shipped defaults suit neither. Extraction, classification, routing, schemas: temperature 0; top_p 1, top_k 0; repetition_penalty 1.0; The same input must give the same output. Writing, brainstorming, variation: temperature 0.7 to 1.0; min_p about 0.05, or top_p 0.9; repetition_penalty 1.05 to 1.1 is defensible; Reasoning models: follow the card, often 0.6. Most local interfaces start at temperature 0.8, top_p 0.95 and a repetition penalty near 1.1. Quotes, colons, commas and braces are what JSON is made of, so that penalty is arguing against correct syntax by the fifth field.\n\n## Settings that work\n\n**Extraction, classification, routing, schema-filling.** Temperature 0, top_p 1, top_k 0, repetition penalty 1.0. You want the same input to give the same output, and you want the model's best guess rather than a sample from its uncertainty.\n\n**Factual question answering over provided context.** Temperature 0 to 0.2. Same reasoning.\n\n**Writing, brainstorming, variation.** Temperature 0.7 to 1.0 with min_p around 0.05, or top_p 0.9. A repetition penalty of 1.05 to 1.1 is defensible here, where the format is prose.\n\n**Reasoning models.** Follow the model card, and note that several of them explicitly recommend *against* temperature 0: often something like temperature 0.6 with top_p 0.95. Greedy decoding on a long chain of thought is prone to getting stuck in loops, because one bad deterministic step repeats forever. This surprises people who have learned that zero is the safe default.\n\n## Determinism\n\nTemperature 0 gives you the same token sequence for the same input on the same machine with the same batch composition. It does not guarantee bit-identical results across different hardware, different batch sizes, or different library versions, because floating-point addition is not associative and kernels reduce in different orders.\n\nFor evaluation this rarely matters, since the differences are tiny and only occasionally flip a near-tie. For a regression test that asserts exact strings it matters a great deal, and the fix is to assert on the parsed result rather than on the raw text.\n\n## The rule to carry away\n\n**Record the sampler with every result.** Two comparisons of the same models can disagree entirely because one was run at temperature 0.8 and the other at 0. It is three numbers in a log line, and it turns an argument into a fact.\n\n```json\n{\"model\":\"qwen3-8b\",\"rev\":\"9a3f21c\",\"quant\":\"Q4_K_M\",\n \"temperature\":0,\"top_p\":1,\"top_k\":0,\"repeat_penalty\":1.0,\"max_tokens\":512}\n```"},"check":{"question":"A model producing JSON starts emitting malformed output after the fourth field (missing quotes, dropped commas) but produces valid JSON for short objects. What should you suspect first?","options":[{"text":"A repetition penalty above 1.0 is progressively suppressing the punctuation tokens the format requires.","correct":true},{"text":"The context window is being exceeded, truncating the later fields.","correct":false,"tempting":"Truncation cuts output off entirely rather than corrupting punctuation while continuing to produce fields."},{"text":"Temperature is too high, so the model is choosing unlikely tokens.","correct":false,"tempting":"High temperature would corrupt output from the start rather than degrade progressively with each repeated symbol."},{"text":"The model is too small to hold a schema in mind for more than a few fields.","correct":false,"tempting":"A plausible-sounding capacity story, but it would not explain why short objects are perfect and the degradation tracks repetition."}]},"prev":{"slug":"models-on-phones","title":"What actually runs on a phone","url":"https://addaly.com/learn/open-models/models-on-phones"},"next":{"slug":"structured-output","title":"Making the output valid by construction","url":"https://addaly.com/learn/open-models/structured-output"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'The sampler, and the defaults that ship broken', in The Open Model Ecosystem. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/open-models/sampling-parameters (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"The sampler, and the defaults that ship broken","courseTitle":"The Open Model Ecosystem","module":{"index":5,"slug":"getting-good-output","title":"Getting good output"},"answer":null,"keyIdea":"The sampler converts logits into a token, and the common default of a repetition penalty above 1 actively damages JSON and code because those formats require repeated punctuation tokens.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure 5.1","figure":{"left":{"title":"Extraction, classification, routing, schemas","items":["temperature 0","top_p 1, top_k 0","repetition_penalty 1.0","The same input must give the same output"]},"type":"compare","right":{"title":"Writing, brainstorming, variation","items":["temperature 0.7 to 1.0","min_p about 0.05, or top_p 0.9","repetition_penalty 1.05 to 1.1 is defensible","Reasoning models: follow the card, often 0.6"]},"title":"Two sets of sampler settings, and why the shipped defaults suit neither","caption":"Most local interfaces start at temperature 0.8, top_p 0.95 and a repetition penalty near 1.1. Quotes, colons, commas and braces are what JSON is made of, so that penalty is arguing against correct syntax by the fifth field."},"alt":"Figure: Two sets of sampler settings, and why the shipped defaults suit neither. Extraction, classification, routing, schemas: temperature 0; top_p 1, top_k 0; repetition_penalty 1.0; The same input must give the same output. Writing, brainstorming, variation: temperature 0.7 to 1.0; min_p about 0.05, or top_p 0.9; repetition_penalty 1.05 to 1.1 is defensible; Reasoning models: follow the card, often 0.6. Most local interfaces start at temperature 0.8, top_p 0.95 and a repetition penalty near 1.1. Quotes, colons, commas and braces are what JSON is made of, so that penalty is arguing against correct syntax by the fifth field."}},"blocks":[{"t":"h2","id":"what-happens-after-the-forward-pass","runs":[{"text":"What happens after the forward pass"}]},{"t":"p","runs":[{"text":"The model outputs a logit for every token in the vocabulary: 150,000 raw scores. The sampler turns those into one chosen token, and every parameter you have heard of is a step in that conversion."}]},{"t":"p","runs":[{"text":"Temperature","b":true},{"text":" divides the logits before the softmax. Below 1 the distribution sharpens and the likeliest tokens dominate; above 1 it flattens and unlikely tokens get a chance. At 0 most implementations skip sampling entirely and take the highest-scoring token, which is greedy decoding."}]},{"t":"p","runs":[{"text":"top_k","b":true},{"text":" keeps only the k highest-scoring tokens and renormalises. Blunt: k=40 is far too many when the model is certain and far too few when it is uncertain across a wide field."}]},{"t":"p","runs":[{"text":"top_p","b":true},{"text":", or nucleus sampling, keeps the smallest set of tokens whose probabilities sum to p. Adaptive in the right direction: a confident step keeps two tokens, an open one keeps forty."}]},{"t":"p","runs":[{"text":"min_p","b":true},{"text":" keeps tokens whose probability is at least some fraction of the top token’s probability. At min_p 0.05, if the best token has probability 0.6, nothing below 0.03 survives. It behaves better than top_p at high temperature, which is why creative-writing setups have converged on it."}]},{"t":"p","runs":[{"text":"repetition_penalty","b":true},{"text":" divides the logits of tokens that have already appeared, with values above 1 discouraging them. "},{"text":"frequency_penalty","b":true},{"text":" and "},{"text":"presence_penalty","b":true},{"text":" do a similar job additively."}]},{"t":"h2","id":"the-default-that-quietly-ruins-structured-output","runs":[{"text":"The default that quietly ruins structured output"}]},{"t":"p","runs":[{"text":"Most local interfaces ship with something like temperature 0.8, top_p 0.95 and a repetition penalty around 1.1. That is a reasonable configuration for chat and a bad one for everything else, and the repetition penalty is the actively harmful part."}]},{"t":"p","runs":[{"text":"Consider what it does to JSON. The tokens "},{"text":"\"","code":true},{"text":", "},{"text":":","code":true},{"text":", "},{"text":",","code":true},{"text":", "},{"text":"{","code":true},{"text":" and "},{"text":"}","code":true},{"text":" must repeat: that is what the format is. A repetition penalty pushes their logits down every time they appear, so by the fifth field the model is being actively discouraged from producing correct syntax. The same applies to code, where indentation and brackets repeat by necessity, and to any tabular or list output."}]},{"t":"p","runs":[{"text":"Set repetition penalty to 1.0 for structured output.","b":true},{"text":" If the model loops, the cause is usually the prompt or a missing stop token, and the penalty is treating the symptom by damaging the format."}]},{"t":"h2","id":"settings-that-work","runs":[{"text":"Settings that work"}]},{"t":"p","runs":[{"text":"Extraction, classification, routing, schema-filling.","b":true},{"text":" Temperature 0, top_p 1, top_k 0, repetition penalty 1.0. You want the same input to give the same output, and you want the model’s best guess rather than a sample from its uncertainty."}]},{"t":"p","runs":[{"text":"Factual question answering over provided context.","b":true},{"text":" Temperature 0 to 0.2. Same reasoning."}]},{"t":"p","runs":[{"text":"Writing, brainstorming, variation.","b":true},{"text":" Temperature 0.7 to 1.0 with min_p around 0.05, or top_p 0.9. A repetition penalty of 1.05 to 1.1 is defensible here, where the format is prose."}]},{"t":"p","runs":[{"text":"Reasoning models.","b":true},{"text":" Follow the model card, and note that several of them explicitly recommend "},{"text":"against","i":true},{"text":" temperature 0: often something like temperature 0.6 with top_p 0.95. Greedy decoding on a long chain of thought is prone to getting stuck in loops, because one bad deterministic step repeats forever. This surprises people who have learned that zero is the safe default."}]},{"t":"h2","id":"determinism","runs":[{"text":"Determinism"}]},{"t":"p","runs":[{"text":"Temperature 0 gives you the same token sequence for the same input on the same machine with the same batch composition. It does not guarantee bit-identical results across different hardware, different batch sizes, or different library versions, because floating-point addition is not associative and kernels reduce in different orders."}]},{"t":"p","runs":[{"text":"For evaluation this rarely matters, since the differences are tiny and only occasionally flip a near-tie. For a regression test that asserts exact strings it matters a great deal, and the fix is to assert on the parsed result rather than on the raw text."}]},{"t":"h2","id":"the-rule-to-carry-away","runs":[{"text":"The rule to carry away"}]},{"t":"p","runs":[{"text":"Record the sampler with every result.","b":true},{"text":" Two comparisons of the same models can disagree entirely because one was run at temperature 0.8 and the other at 0. It is three numbers in a log line, and it turns an argument into a fact."}]},{"t":"code","lang":"json","label":"JSON","text":"{\"model\":\"qwen3-8b\",\"rev\":\"9a3f21c\",\"quant\":\"Q4_K_M\",\n \"temperature\":0,\"top_p\":1,\"top_k\":0,\"repeat_penalty\":1.0,\"max_tokens\":512}"}],"check":{"question":[{"text":"A model producing JSON starts emitting malformed output after the fourth field (missing quotes, dropped commas) but produces valid JSON for short objects. What should you suspect first?"}],"options":[{"text":[{"text":"A repetition penalty above 1.0 is progressively suppressing the punctuation tokens the format requires."}],"correct":true,"tempting":null},{"text":[{"text":"The context window is being exceeded, truncating the later fields."}],"correct":false,"tempting":[{"text":"Truncation cuts output off entirely rather than corrupting punctuation while continuing to produce fields."}]},{"text":[{"text":"Temperature is too high, so the model is choosing unlikely tokens."}],"correct":false,"tempting":[{"text":"High temperature would corrupt output from the start rather than degrade progressively with each repeated symbol."}]},{"text":[{"text":"The model is too small to hold a schema in mind for more than a few fields."}],"correct":false,"tempting":[{"text":"A plausible-sounding capacity story, but it would not explain why short objects are perfect and the degradation tracks repetition."}]}],"order":[2,1,3,0],"why":null},"prev":{"slug":"models-on-phones","title":"What actually runs on a phone","minutes":8},"next":{"slug":"structured-output","title":"Making the output valid by construction","minutes":9}}}