{"course":{"slug":"how-llms-work","title":"How a Language Model Actually Works","url":"https://addaly.com/learn/how-llms-work","level":"intermediate","track":"start"},"lesson":{"slug":"temperature-and-sampling","title":"Temperature and sampling: what the knob really does","url":"https://addaly.com/learn/how-llms-work/temperature-and-sampling","position":22,"of":78,"minutes":6,"answer":"Temperature reshapes a distribution the model already produced; it cannot add knowledge the model lacks.","keyIdea":"Temperature reshapes a distribution the model already produced; it cannot add knowledge the model lacks."},"body":{"format":"markdown","text":"## From scores to a choice\n\nAt the end of every forward pass you have one raw score, a logit, for each vocabulary entry. Suppose three candidates score 4.0, 3.0 and 1.0 and everything else is far below.\n\nThose numbers are not probabilities. Softmax turns them into probabilities by exponentiating and normalising. For our three candidates that gives roughly:\n\n| candidate | logit | probability |\n|---|---|---|\n| A | 4.0 | 71% |\n| B | 3.0 | 26% |\n| C | 1.0 | 4% |\n\nOne is then drawn at random according to those probabilities. That is sampling, and every knob you have adjusts either the distribution or which parts of it are eligible.\n\n## Temperature\n\nTemperature divides the logits before the softmax. That is all it is.\n\nAt T = 0.5 the logits become 8, 6 and 2, and the same three candidates come out near 88%, 12% and 0.2%. At T = 2.0 they become 2, 1.5 and 0.5, giving roughly 55%, 33% and 12%.\n\nSo low temperature exaggerates whatever gap already existed, and high temperature flattens it, handing unlikely tokens a real chance. T = 0 is the limiting case: always take the highest-scoring token, no randomness at all. Above about 1.5 most models produce visible incoherence, because a token that the model rated at 1% is now being chosen several times per paragraph, and one wrong token poisons everything that follows it.\n\n## Top-k and top-p\n\nTemperature alone has an awkward property: it raises the odds of every unlikely token, including the absurd ones.\n\n> Figure: Three candidates, one knob. Line chart of Probability, % against Temperature: A, logit 4.0 (0.25, 98.2) (0.5, 87.9) (1, 70.5) (1.5, 60.6) (2, 54.7); B, logit 3.0 (0.25, 1.8) (0.5, 11.9) (1, 26) (1.5, 31.1) (2, 33.2); C, logit 1.0 (0.25, 0) (0.5, 0.2) (1, 3.5) (1.5, 8.2) (2, 12.2). Logits of 4.0, 3.0 and 1.0, divided by the temperature before the softmax. That division is the whole of what temperature does: it exaggerates or flattens a gap the model already produced, and it cannot add a candidate the model never scored.\n\nTop-k fixes this crudely: keep only the k highest-scoring candidates and renormalise. Top-p, also called nucleus sampling, is smarter: keep the smallest set of candidates whose probabilities add up to p, typically 0.9 or 0.95. When the model is confident the set may hold two tokens; when it is unsure it may hold two hundred. The cut adapts to the distribution rather than fixing a count.\n\nMost systems apply top-p and temperature together. Tune one at a time or you will not be able to tell which did what.\n\n## Two honest corrections\n\n**T = 0 is not \"accurate mode\".** It gives you the model's single most likely continuation. If the model's most likely continuation is wrong, T = 0 delivers that wrong answer every time, with the confidence of the top of the distribution behind it. Determinism is not accuracy. What T = 0 buys you is reproducibility, which matters for tests and for debugging, and that is the honest reason to use it.\n\n**T = 0 is not even fully deterministic in practice.** On production hardware, requests are batched, and floating-point addition is not associative: the order in which numbers are summed depends on batch composition and kernel choices, so two logits separated by a hair can swap places between runs. Providers have shipped work to reduce this, but if you have written a test that asserts an exact string from a hosted model, expect it to fail occasionally for reasons that have nothing to do with your code.\n\n## Choosing a setting\n\n- Extraction, classification, structured output, anything you compare against an expected value: T near 0.\n- Ordinary assistant work, explanation, summaries: the provider default, usually near 0.7 with top-p around 0.95.\n- Naming, brainstorming, first drafts: T around 1.0, and generate several, rather than pushing to 1.6 and reading wreckage.\n\nOne more use worth knowing. Sample the same question five times at moderate temperature and compare. If the five answers agree, the model is on solid ground. If they contradict each other, you have a cheap uncertainty signal that the model's own confident tone will never give you."},"check":{"question":"A support bot gives a confidently wrong answer about a refund policy. A colleague suggests setting temperature to 0 to fix it. What should you expect?","options":[{"text":"Accuracy improves, because the randomness that caused the error has been removed.","correct":false,"tempting":"Randomness is the most visible difference between runs, so it looks like the obvious culprit for any output that came out wrong."},{"text":"You get the model's single most likely answer, which is the same wrong one, now produced consistently every time.","correct":true},{"text":"The bot starts declining to answer rather than guessing, since low temperature makes it conservative.","correct":false,"tempting":"\"Conservative\" is the word everyone uses for low temperature, and it sounds like caution about making claims."},{"text":"Nothing changes, because temperature affects wording and not content.","correct":false,"tempting":"Temperature is usually demonstrated on style (the same answer phrased differently) so it looks purely cosmetic."}]},"prev":{"slug":"greedy-beam-and-degeneration","title":"Greedy, beam search, and why chat models refuse both","url":"https://addaly.com/learn/how-llms-work/greedy-beam-and-degeneration"},"next":{"slug":"reading-logprobs","title":"Reading the numbers the model will give you","url":"https://addaly.com/learn/how-llms-work/reading-logprobs"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Temperature and sampling: what the knob really does', in How a Language Model Actually Works. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/how-llms-work/temperature-and-sampling (Accessed: 31 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Temperature and sampling: what the knob really does","courseTitle":"How a Language Model Actually Works","module":{"index":3,"slug":"turning-scores-into-text","title":"Turning scores into text"},"answer":null,"keyIdea":"Temperature reshapes a distribution the model already produced; it cannot add knowledge the model lacks.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"type":"line","title":"Three candidates, one knob","series":[{"label":"A, logit 4.0","points":[[0.25,98.2],[0.5,87.9],[1,70.5],[1.5,60.6],[2,54.7]]},{"label":"B, logit 3.0","points":[[0.25,1.8],[0.5,11.9],[1,26],[1.5,31.1],[2,33.2]]},{"label":"C, logit 1.0","points":[[0.25,0],[0.5,0.2],[1,3.5],[1.5,8.2],[2,12.2]]}],"xLabel":"Temperature","yLabel":"Probability, %","caption":"Logits of 4.0, 3.0 and 1.0, divided by the temperature before the softmax. That division is the whole of what temperature does: it exaggerates or flattens a gap the model already produced, and it cannot add a candidate the model never scored."},"alt":"Figure: Three candidates, one knob. Line chart of Probability, % against Temperature: A, logit 4.0 (0.25, 98.2) (0.5, 87.9) (1, 70.5) (1.5, 60.6) (2, 54.7); B, logit 3.0 (0.25, 1.8) (0.5, 11.9) (1, 26) (1.5, 31.1) (2, 33.2); C, logit 1.0 (0.25, 0) (0.5, 0.2) (1, 3.5) (1.5, 8.2) (2, 12.2). Logits of 4.0, 3.0 and 1.0, divided by the temperature before the softmax. That division is the whole of what temperature does: it exaggerates or flattens a gap the model already produced, and it cannot add a candidate the model never scored."}},"blocks":[{"t":"h2","id":"from-scores-to-a-choice","runs":[{"text":"From scores to a choice"}]},{"t":"p","runs":[{"text":"At the end of every forward pass you have one raw score, a logit, for each vocabulary entry. Suppose three candidates score 4.0, 3.0 and 1.0 and everything else is far below."}]},{"t":"p","runs":[{"text":"Those numbers are not probabilities. Softmax turns them into probabilities by exponentiating and normalising. For our three candidates that gives roughly:"}]},{"t":"table","head":[[{"text":"candidate"}],[{"text":"logit"}],[{"text":"probability"}]],"align":["left","left","left"],"rows":[[[{"text":"A"}],[{"text":"4.0"}],[{"text":"71%"}]],[[{"text":"B"}],[{"text":"3.0"}],[{"text":"26%"}]],[[{"text":"C"}],[{"text":"1.0"}],[{"text":"4%"}]]]},{"t":"p","runs":[{"text":"One is then drawn at random according to those probabilities. That is sampling, and every knob you have adjusts either the distribution or which parts of it are eligible."}]},{"t":"h2","id":"temperature","runs":[{"text":"Temperature"}]},{"t":"p","runs":[{"text":"Temperature divides the logits before the softmax. That is all it is."}]},{"t":"p","runs":[{"text":"At T = 0.5 the logits become 8, 6 and 2, and the same three candidates come out near 88%, 12% and 0.2%. At T = 2.0 they become 2, 1.5 and 0.5, giving roughly 55%, 33% and 12%."}]},{"t":"p","runs":[{"text":"So low temperature exaggerates whatever gap already existed, and high temperature flattens it, handing unlikely tokens a real chance. T = 0 is the limiting case: always take the highest-scoring token, no randomness at all. Above about 1.5 most models produce visible incoherence, because a token that the model rated at 1% is now being chosen several times per paragraph, and one wrong token poisons everything that follows it."}]},{"t":"h2","id":"top-k-and-top-p","runs":[{"text":"Top-k and top-p"}]},{"t":"p","runs":[{"text":"Temperature alone has an awkward property: it raises the odds of every unlikely token, including the absurd ones."}]},{"t":"p","runs":[{"text":"Top-k fixes this crudely: keep only the k highest-scoring candidates and renormalise. Top-p, also called nucleus sampling, is smarter: keep the smallest set of candidates whose probabilities add up to p, typically 0.9 or 0.95. When the model is confident the set may hold two tokens; when it is unsure it may hold two hundred. The cut adapts to the distribution rather than fixing a count."}]},{"t":"p","runs":[{"text":"Most systems apply top-p and temperature together. Tune one at a time or you will not be able to tell which did what."}]},{"t":"h2","id":"two-honest-corrections","runs":[{"text":"Two honest corrections"}]},{"t":"p","runs":[{"text":"T = 0 is not “accurate mode”.","b":true},{"text":" It gives you the model’s single most likely continuation. If the model’s most likely continuation is wrong, T = 0 delivers that wrong answer every time, with the confidence of the top of the distribution behind it. Determinism is not accuracy. What T = 0 buys you is reproducibility, which matters for tests and for debugging, and that is the honest reason to use it."}]},{"t":"p","runs":[{"text":"T = 0 is not even fully deterministic in practice.","b":true},{"text":" On production hardware, requests are batched, and floating-point addition is not associative: the order in which numbers are summed depends on batch composition and kernel choices, so two logits separated by a hair can swap places between runs. Providers have shipped work to reduce this, but if you have written a test that asserts an exact string from a hosted model, expect it to fail occasionally for reasons that have nothing to do with your code."}]},{"t":"h2","id":"choosing-a-setting","runs":[{"text":"Choosing a setting"}]},{"t":"ul","items":[[{"text":"Extraction, classification, structured output, anything you compare against an expected value: T near 0."}],[{"text":"Ordinary assistant work, explanation, summaries: the provider default, usually near 0.7 with top-p around 0.95."}],[{"text":"Naming, brainstorming, first drafts: T around 1.0, and generate several, rather than pushing to 1.6 and reading wreckage."}]]},{"t":"p","runs":[{"text":"One more use worth knowing. Sample the same question five times at moderate temperature and compare. If the five answers agree, the model is on solid ground. If they contradict each other, you have a cheap uncertainty signal that the model’s own confident tone will never give you."}]}],"check":{"question":[{"text":"A support bot gives a confidently wrong answer about a refund policy. A colleague suggests setting temperature to 0 to fix it. What should you expect?"}],"options":[{"text":[{"text":"Accuracy improves, because the randomness that caused the error has been removed."}],"correct":false,"tempting":[{"text":"Randomness is the most visible difference between runs, so it looks like the obvious culprit for any output that came out wrong."}]},{"text":[{"text":"You get the model’s single most likely answer, which is the same wrong one, now produced consistently every time."}],"correct":true,"tempting":null},{"text":[{"text":"The bot starts declining to answer rather than guessing, since low temperature makes it conservative."}],"correct":false,"tempting":[{"text":"“Conservative” is the word everyone uses for low temperature, and it sounds like caution about making claims."}]},{"text":[{"text":"Nothing changes, because temperature affects wording and not content."}],"correct":false,"tempting":[{"text":"Temperature is usually demonstrated on style (the same answer phrased differently) so it looks purely cosmetic."}]}],"order":[2,0,1,3],"why":null},"prev":{"slug":"greedy-beam-and-degeneration","title":"Greedy, beam search, and why chat models refuse both","minutes":9},"next":{"slug":"reading-logprobs","title":"Reading the numbers the model will give you","minutes":9}}}