From scores to a choice
At the end of every forward pass you have one raw score, a logit, for each vocabulary entry. Suppose three candidates score 4.0, 3.0 and 1.0 and everything else is far below.
Those numbers are not probabilities. Softmax turns them into probabilities by exponentiating and normalising. For our three candidates that gives roughly:
| candidate | logit | probability | |---|---|---| | A | 4.0 | 71% | | B | 3.0 | 26% | | C | 1.0 | 4% |
One is then drawn at random according to those probabilities. That is sampling, and every knob you have adjusts either the distribution or which parts of it are eligible.
Temperature
Temperature divides the logits before the softmax. That is all it is.
At T = 0.5 the logits become 8, 6 and 2, and the same three candidates come out near 88%, 12% and 0.2%. At T = 2.0 they become 2, 1.5 and 0.5, giving roughly 55%, 33% and 12%.
So low temperature exaggerates whatever gap already existed, and high temperature flattens it, handing unlikely tokens a real chance. T = 0 is the limiting case: always take the highest-scoring token, no randomness at all. Above about 1.5 most models produce visible incoherence, because a token that the model rated at 1% is now being chosen several times per paragraph, and one wrong token poisons everything that follows it.
Top-k and top-p
Temperature alone has an awkward property: it raises the odds of every unlikely token, including the genuinely absurd ones.
Top-k fixes this crudely — keep only the k highest-scoring candidates and renormalise. Top-p, also called nucleus sampling, is smarter: keep the smallest set of candidates whose probabilities add up to p, typically 0.9 or 0.95. When the model is confident the set may hold two tokens; when it is genuinely unsure it may hold two hundred. The cut adapts to the distribution rather than fixing a count.
Most systems apply top-p and temperature together. Tune one at a time or you will not be able to tell which did what.
Two honest corrections
T = 0 is not "accurate mode". It gives you the model's single most likely continuation. If the model's most likely continuation is wrong, T = 0 delivers that wrong answer every time, with the confidence of the top of the distribution behind it. Determinism is not accuracy. What T = 0 buys you is reproducibility, which matters for tests and for debugging, and that is the honest reason to use it.
T = 0 is not even fully deterministic in practice. On production hardware, requests are batched, and floating-point addition is not associative — the order in which numbers are summed depends on batch composition and kernel choices, so two logits separated by a hair can swap places between runs. Providers have shipped work to reduce this, but if you have written a test that asserts an exact string from a hosted model, expect it to fail occasionally for reasons that have nothing to do with your code.
Choosing a setting
- Extraction, classification, structured output, anything you compare against an expected value: T near 0.
- Ordinary assistant work, explanation, summaries: the provider default, usually near 0.7 with top-p around 0.95.
- Naming, brainstorming, first drafts: T around 1.0, and generate several, rather than pushing to 1.6 and reading wreckage.
One more use worth knowing. Sample the same question five times at moderate temperature and compare. If the five answers agree, the model is on solid ground. If they contradict each other, you have a cheap uncertainty signal that the model's own confident tone will never give you.
Before you move on