The sampler, and the defaults that ship broken

What happens after the forward pass

The model outputs a logit for every token in the vocabulary: 150,000 raw scores. The sampler turns those into one chosen token, and every parameter you have heard of is a step in that conversion.

Temperature divides the logits before the softmax. Below 1 the distribution sharpens and the likeliest tokens dominate; above 1 it flattens and unlikely tokens get a chance. At 0 most implementations skip sampling entirely and take the highest-scoring token, which is greedy decoding.

top_k keeps only the k highest-scoring tokens and renormalises. Blunt: k=40 is far too many when the model is certain and far too few when it is uncertain across a wide field.

top_p, or nucleus sampling, keeps the smallest set of tokens whose probabilities sum to p. Adaptive in the right direction: a confident step keeps two tokens, an open one keeps forty.

min_p keeps tokens whose probability is at least some fraction of the top token’s probability. At min_p 0.05, if the best token has probability 0.6, nothing below 0.03 survives. It behaves better than top_p at high temperature, which is why creative-writing setups have converged on it.

repetition_penalty divides the logits of tokens that have already appeared, with values above 1 discouraging them. frequency_penalty and presence_penalty do a similar job additively.

The default that quietly ruins structured output

Most local interfaces ship with something like temperature 0.8, top_p 0.95 and a repetition penalty around 1.1. That is a reasonable configuration for chat and a bad one for everything else, and the repetition penalty is the actively harmful part.

Consider what it does to JSON. The tokens ", :, ,, { and } must repeat: that is what the format is. A repetition penalty pushes their logits down every time they appear, so by the fifth field the model is being actively discouraged from producing correct syntax. The same applies to code, where indentation and brackets repeat by necessity, and to any tabular or list output.

Set repetition penalty to 1.0 for structured output. If the model loops, the cause is usually the prompt or a missing stop token, and the penalty is treating the symptom by damaging the format.

Settings that work

Extraction, classification, routing, schema-filling. Temperature 0, top_p 1, top_k 0, repetition penalty 1.0. You want the same input to give the same output, and you want the model’s best guess rather than a sample from its uncertainty.

Factual question answering over provided context. Temperature 0 to 0.2. Same reasoning.

Writing, brainstorming, variation. Temperature 0.7 to 1.0 with min_p around 0.05, or top_p 0.9. A repetition penalty of 1.05 to 1.1 is defensible here, where the format is prose.

Reasoning models. Follow the model card, and note that several of them explicitly recommend against temperature 0: often something like temperature 0.6 with top_p 0.95. Greedy decoding on a long chain of thought is prone to getting stuck in loops, because one bad deterministic step repeats forever. This surprises people who have learned that zero is the safe default.

Determinism

Temperature 0 gives you the same token sequence for the same input on the same machine with the same batch composition. It does not guarantee bit-identical results across different hardware, different batch sizes, or different library versions, because floating-point addition is not associative and kernels reduce in different orders.

For evaluation this rarely matters, since the differences are tiny and only occasionally flip a near-tie. For a regression test that asserts exact strings it matters a great deal, and the fix is to assert on the parsed result rather than on the raw text.

The rule to carry away

Record the sampler with every result. Two comparisons of the same models can disagree entirely because one was run at temperature 0.8 and the other at 0. It is three numbers in a log line, and it turns an argument into a fact.

{"model":"qwen3-8b","rev":"9a3f21c","quant":"Q4_K_M",
 "temperature":0,"top_p":1,"top_k":0,"repeat_penalty":1.0,"max_tokens":512}