Sampling: why output quality is often not the model

A model does not produce text. It produces a probability distribution over the next token, about 32,000 to 150,000 numbers wide, and something else chooses. That chooser is the sampler, and a surprising share of "this model is bad" is a sampler set wrong.

The pipeline

Logits come out of the model, then, in most backends: penalties are applied, temperature scales, truncation methods cut the tail, and one token is drawn from what remains. The order varies between backends and occasionally matters.

temperature divides the logits before softmax. Below 1 sharpens the distribution toward the top token; above 1 flattens it. At 0 it is greedy — always the most likely token, fully reproducible.

top-k keeps the k highest-probability tokens. Blunt: it keeps 40 candidates whether the model is certain or lost.

top-p (nucleus) keeps the smallest set of tokens whose probabilities sum to p. Adaptive, and the default almost everywhere. Its failure mode is the flat distribution: when the model is genuinely uncertain, p=0.95 can admit several hundred tokens, most of them bad.

min-p keeps tokens whose probability is at least min_p x p_max. This scales with the model's own confidence — narrow when it is sure, wide when it is not — which is exactly the behaviour top-p fails at. min_p 0.05 with temperature 1.0 is a good modern default, and it tolerates high temperature far better.

repetition_penalty divides the logits of tokens seen in the last N tokens. It is token-blind, so it also penalises }, def, the, and every newline. Above about 1.15 you can watch it break indentation and drop closing brackets.

DRY (Don't Repeat Yourself) penalises only tokens that would *continue an already-repeated sequence*, with the penalty growing with match length. It stops loops without punishing the ordinary repetition that structured text requires. Usual settings: multiplier 0.8, base 1.75, allowed length 2.

XTC (Exclude Top Choices) sometimes removes all but the least likely of the tokens above a probability threshold — deliberately throwing away the obvious continuation. It makes prose less predictable and makes anything that must be correct worse. Use it for fiction, never for code or extraction.

Settings by task

Extraction, classification, code, JSON: temperature 0, or 0.2 with top_p 1.0. No repetition penalty. If your backend supports constrained decoding, use it — llama.cpp --grammar, vLLM guided decoding, SGLang's grammar backend. A schema is stronger than any prompt instruction.

General assistant: temperature 0.7 with min_p 0.05, no repetition penalty. Or better, whatever the model card recommends — many now specify exact numbers, and those were tuned against the model's own training.

Creative writing: temperature 1.0–1.2, min_p 0.05, DRY on, XTC optional.

curl http://localhost:8080/v1/chat/completions -H 'Content-Type: application/json' -d '{
  "model": "local",
  "messages": [{"role": "user", "content": "Return only JSON: {\"city\": ...}"}],
  "temperature": 0,
  "top_p": 1.0
}'

Check these before touching the sampler

Most bad output is not a sampler problem either. In rough order of frequency:

1. 1Wrong chat template. The model was trained on specific role markers and is not seeing them. Output is subtly off in a way that looks like low intelligence.
2. 2Silent context truncation. Lesson 4.
3. 3You loaded the base model, not the instruct model. Base models continue text; they do not answer.
4. 4A system prompt you forgot about, injected by your UI.

Work down that list first. Then change one sampler value at a time, with a fixed seed and the same five prompts, and read the outputs. Changing four knobs at once and forming an impression is how people end up with settings they cannot explain and will not abandon.