{"course":{"slug":"hugging-face","title":"Hugging Face, End to End","url":"https://addaly.com/learn/hugging-face","level":"intermediate","track":"create"},"lesson":{"slug":"from-logits-to-text","title":"Logits, and the four knobs between them and words","url":"https://addaly.com/learn/hugging-face/from-logits-to-text","position":14,"of":75,"minutes":10,"answer":"A model outputs one unnormalised score per vocabulary token and the decoding strategy picks from them, so temperature, top-p and repetition penalties are choices you make after the model has finished.","keyIdea":"A model outputs one unnormalised score per vocabulary token and the decoding strategy picks from them, so temperature, top-p and repetition penalties are choices you make after the model has finished; and a `do_sample: false` default in generation_config silently discards every sampling parameter you pass."},"body":{"format":"markdown","text":"## What comes out of the model\n\n```python\nout = model(**enc)\nprint(out.logits.shape)   # [batch, sequence_length, vocab_size]\n```\n\nFor a causal model with a 151,936-token vocabulary and a 12-token prompt, that is a tensor of 12 × 151,936 raw numbers. Only the last row matters for generating the next token: it holds one score per possible next token, unnormalised, typically between about −20 and +20.\n\nSoftmax turns that row into a probability distribution. Then something has to **choose**, and everything people call creativity or randomness lives in that choice.\n\n```python\nimport torch\nnext_logits = out.logits[0, -1]\nprobs = torch.softmax(next_logits, dim=-1)\ntop = torch.topk(probs, 5)\nfor p, i in zip(top.values, top.indices):\n    print(f\"{p:.3f}  {tok.decode(i)!r}\")\n```\n\nRun that on a half-finished sentence once. Seeing the actual candidates, and how often the top one holds 0.95 of the mass, makes every knob below intuitive rather than mystical.\n\n## Greedy, and why it repeats\n\nGreedy decoding takes the argmax every step. It is deterministic, which is exactly what you want for classification-like tasks, extraction and anything you need to reproduce.\n\nIt also falls into loops. Once the model emits a phrase, that phrase raises the probability of itself recurring, and with no randomness nothing breaks the cycle, so you get *\"the results show that the results show that the results show that\"*. This is a property of maximising at every step, not a defect in a particular model.\n\n```python\nout = model.generate(**enc, do_sample=False, max_new_tokens=200)\n```\n\n## Temperature\n\nTemperature divides the logits before the softmax. Below 1 it sharpens the distribution: the likely tokens get likelier. Above 1 it flattens it. At 0 it is greedy.\n\n- `0.0–0.3`: extraction, classification, code, anything with a right answer.\n- `0.7–0.8`: general assistant use. Most chat defaults sit here.\n- `1.0–1.3`: brainstorming, variety, fiction.\n- Above `1.5`: usually incoherent, because low-probability tokens become reachable.\n\n## Top-k and top-p\n\nBoth truncate the candidate list before sampling.\n\n- **top-k** keeps the k highest-probability tokens. Fixed count, so it keeps 50 candidates whether the model is certain or not.\n- **top-p** (nucleus) keeps the smallest set whose probabilities sum to p. Adaptive: when the model is confident, that might be two tokens; when it is unsure, forty.\n\ntop-p is usually the better default precisely because it responds to the model's confidence. `top_p=0.9` with `temperature=0.7` is the combination behind most defaults you will meet.\n\n## Repetition controls\n\n`repetition_penalty=1.1` divides the logits of tokens already present. `no_repeat_ngram_size=3` forbids any 3-gram from occurring twice. The second is blunt: it makes a model literally unable to repeat a person's name three times, which in a summary of a legal document is wrong. Prefer a mild repetition penalty and fix the rest with the prompt.\n\n## The interaction that wastes an afternoon\n\n```python\nout = model.generate(**enc, temperature=0.9, top_p=0.95, max_new_tokens=100)\n```\n\nIf the model's `generation_config.json` has `do_sample: false`, this runs **greedily** and your temperature does nothing. Recent versions warn; older ones do not. The output is identical on every run and people conclude the model is broken or the seed is stuck.\n\nBe explicit:\n\n```python\nout = model.generate(**enc, do_sample=True, temperature=0.9, top_p=0.95,\n                     max_new_tokens=100)\n```\n\nInspect what you inherited with `print(model.generation_config)` before benchmarking anything. Two models that appear to differ in quality frequently differ only here.\n\n## Decoding back\n\n```python\ntext = tok.decode(out[0], skip_special_tokens=True)\n```\n\n`generate` returns the prompt plus the continuation, so trimming matters:\n\n```python\nnew_tokens = out[0][enc[\"input_ids\"].shape[-1]:]\nprint(tok.decode(new_tokens, skip_special_tokens=True))\n```\n\nWithout `skip_special_tokens=True` you see `<|im_end|>` and friends in your UI, which is the single most common cosmetic bug in a first demo.\n\n## Reproducibility\n\n`set_seed(0)` makes sampling repeatable on the same hardware and library versions. Across GPUs, dtypes or versions, floating-point accumulation order differs and the same seed can produce different text. Determinism is per-configuration, not absolute, and any evaluation that assumes otherwise will chase ghosts.\n\n## Do this now\n\nGenerate the same prompt four times: greedy, then `temperature=0.7, top_p=0.9`, then `temperature=1.5`, then greedy with `no_repeat_ngram_size=3`. Four runs, one prompt. That comparison teaches more about decoding than any explanation."},"check":{"question":"A developer sets `temperature=1.2` and gets byte-identical output on every run. What is the most likely cause?","options":[{"text":"The random seed is fixed somewhere in the environment, so sampling is reproducible rather than disabled.","correct":false,"tempting":"Would explain identical output, but a fixed seed is an unusual default and does not explain output that looks deterministic and safe."},{"text":"The model's generation config sets `do_sample` to false, so decoding is greedy and the temperature is ignored.","correct":true},{"text":"Temperature above 1.0 is clamped back to 1.0 by the library, which makes the distribution effectively deterministic for confident models.","correct":false,"tempting":"Invents a clamp; values above 1 flatten the distribution and increase variety rather than removing it."},{"text":"Sampling only applies once `top_p` or `top_k` is also set, because temperature alone has no candidate set to sample from.","correct":false,"tempting":"Assumes a dependency between parameters; temperature alone still defines a full distribution to sample from."}]},"prev":{"slug":"padding-and-attention-masks","title":"Padding on the wrong side ruins generation quietly","url":"https://addaly.com/learn/hugging-face/padding-and-attention-masks"},"next":{"slug":"chat-templates","title":"The chat template is not optional","url":"https://addaly.com/learn/hugging-face/chat-templates"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Logits, and the four knobs between them and words', in Hugging Face, End to End. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/hugging-face/from-logits-to-text (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Logits, and the four knobs between them and words","courseTitle":"Hugging Face, End to End","module":{"index":2,"slug":"transformers-from-text-to-text","title":"Transformers, from characters to output"},"answer":null,"keyIdea":"A model outputs one unnormalised score per vocabulary token and the decoding strategy picks from them, so temperature, top-p and repetition penalties are choices you make after the model has finished; and a `do_sample: false` default in generation_config silently discards every sampling parameter you pass.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"what-comes-out-of-the-model","runs":[{"text":"What comes out of the model"}]},{"t":"code","lang":"python","label":"Python","text":"out = model(**enc)\nprint(out.logits.shape)   # [batch, sequence_length, vocab_size]"},{"t":"p","runs":[{"text":"For a causal model with a 151,936-token vocabulary and a 12-token prompt, that is a tensor of 12 × 151,936 raw numbers. Only the last row matters for generating the next token: it holds one score per possible next token, unnormalised, typically between about −20 and +20."}]},{"t":"p","runs":[{"text":"Softmax turns that row into a probability distribution. Then something has to "},{"text":"choose","b":true},{"text":", and everything people call creativity or randomness lives in that choice."}]},{"t":"code","lang":"python","label":"Python","text":"import torch\nnext_logits = out.logits[0, -1]\nprobs = torch.softmax(next_logits, dim=-1)\ntop = torch.topk(probs, 5)\nfor p, i in zip(top.values, top.indices):\n    print(f\"{p:.3f}  {tok.decode(i)!r}\")"},{"t":"p","runs":[{"text":"Run that on a half-finished sentence once. Seeing the actual candidates, and how often the top one holds 0.95 of the mass, makes every knob below intuitive rather than mystical."}]},{"t":"h2","id":"greedy-and-why-it-repeats","runs":[{"text":"Greedy, and why it repeats"}]},{"t":"p","runs":[{"text":"Greedy decoding takes the argmax every step. It is deterministic, which is exactly what you want for classification-like tasks, extraction and anything you need to reproduce."}]},{"t":"p","runs":[{"text":"It also falls into loops. Once the model emits a phrase, that phrase raises the probability of itself recurring, and with no randomness nothing breaks the cycle, so you get "},{"text":"”the results show that the results show that the results show that”","i":true},{"text":". This is a property of maximising at every step, not a defect in a particular model."}]},{"t":"code","lang":"python","label":"Python","text":"out = model.generate(**enc, do_sample=False, max_new_tokens=200)"},{"t":"h2","id":"temperature","runs":[{"text":"Temperature"}]},{"t":"p","runs":[{"text":"Temperature divides the logits before the softmax. Below 1 it sharpens the distribution: the likely tokens get likelier. Above 1 it flattens it. At 0 it is greedy."}]},{"t":"ul","items":[[{"text":"0.0–0.3","code":true},{"text":": extraction, classification, code, anything with a right answer."}],[{"text":"0.7–0.8","code":true},{"text":": general assistant use. Most chat defaults sit here."}],[{"text":"1.0–1.3","code":true},{"text":": brainstorming, variety, fiction."}],[{"text":"Above "},{"text":"1.5","code":true},{"text":": usually incoherent, because low-probability tokens become reachable."}]]},{"t":"h2","id":"top-k-and-top-p","runs":[{"text":"Top-k and top-p"}]},{"t":"p","runs":[{"text":"Both truncate the candidate list before sampling."}]},{"t":"ul","items":[[{"text":"top-k","b":true},{"text":" keeps the k highest-probability tokens. Fixed count, so it keeps 50 candidates whether the model is certain or not."}],[{"text":"top-p","b":true},{"text":" (nucleus) keeps the smallest set whose probabilities sum to p. Adaptive: when the model is confident, that might be two tokens; when it is unsure, forty."}]]},{"t":"p","runs":[{"text":"top-p is usually the better default precisely because it responds to the model’s confidence. "},{"text":"top_p=0.9","code":true},{"text":" with "},{"text":"temperature=0.7","code":true},{"text":" is the combination behind most defaults you will meet."}]},{"t":"h2","id":"repetition-controls","runs":[{"text":"Repetition controls"}]},{"t":"p","runs":[{"text":"repetition_penalty=1.1","code":true},{"text":" divides the logits of tokens already present. "},{"text":"no_repeat_ngram_size=3","code":true},{"text":" forbids any 3-gram from occurring twice. The second is blunt: it makes a model literally unable to repeat a person’s name three times, which in a summary of a legal document is wrong. Prefer a mild repetition penalty and fix the rest with the prompt."}]},{"t":"h2","id":"the-interaction-that-wastes-an-afternoon","runs":[{"text":"The interaction that wastes an afternoon"}]},{"t":"code","lang":"python","label":"Python","text":"out = model.generate(**enc, temperature=0.9, top_p=0.95, max_new_tokens=100)"},{"t":"p","runs":[{"text":"If the model’s "},{"text":"generation_config.json","code":true},{"text":" has "},{"text":"do_sample: false","code":true},{"text":", this runs "},{"text":"greedily","b":true},{"text":" and your temperature does nothing. Recent versions warn; older ones do not. The output is identical on every run and people conclude the model is broken or the seed is stuck."}]},{"t":"p","runs":[{"text":"Be explicit:"}]},{"t":"code","lang":"python","label":"Python","text":"out = model.generate(**enc, do_sample=True, temperature=0.9, top_p=0.95,\n                     max_new_tokens=100)"},{"t":"p","runs":[{"text":"Inspect what you inherited with "},{"text":"print(model.generation_config)","code":true},{"text":" before benchmarking anything. Two models that appear to differ in quality frequently differ only here."}]},{"t":"h2","id":"decoding-back","runs":[{"text":"Decoding back"}]},{"t":"code","lang":"python","label":"Python","text":"text = tok.decode(out[0], skip_special_tokens=True)"},{"t":"p","runs":[{"text":"generate","code":true},{"text":" returns the prompt plus the continuation, so trimming matters:"}]},{"t":"code","lang":"python","label":"Python","text":"new_tokens = out[0][enc[\"input_ids\"].shape[-1]:]\nprint(tok.decode(new_tokens, skip_special_tokens=True))"},{"t":"p","runs":[{"text":"Without "},{"text":"skip_special_tokens=True","code":true},{"text":" you see "},{"text":"<|im_end|>","code":true},{"text":" and friends in your UI, which is the single most common cosmetic bug in a first demo."}]},{"t":"h2","id":"reproducibility","runs":[{"text":"Reproducibility"}]},{"t":"p","runs":[{"text":"set_seed(0)","code":true},{"text":" makes sampling repeatable on the same hardware and library versions. Across GPUs, dtypes or versions, floating-point accumulation order differs and the same seed can produce different text. Determinism is per-configuration, not absolute, and any evaluation that assumes otherwise will chase ghosts."}]},{"t":"h2","id":"do-this-now","runs":[{"text":"Do this now"}],"role":"try"},{"t":"p","runs":[{"text":"Generate the same prompt four times: greedy, then "},{"text":"temperature=0.7, top_p=0.9","code":true},{"text":", then "},{"text":"temperature=1.5","code":true},{"text":", then greedy with "},{"text":"no_repeat_ngram_size=3","code":true},{"text":". Four runs, one prompt. That comparison teaches more about decoding than any explanation."}]}],"check":{"question":[{"text":"A developer sets "},{"text":"temperature=1.2","code":true},{"text":" and gets byte-identical output on every run. What is the most likely cause?"}],"options":[{"text":[{"text":"The random seed is fixed somewhere in the environment, so sampling is reproducible rather than disabled."}],"correct":false,"tempting":[{"text":"Would explain identical output, but a fixed seed is an unusual default and does not explain output that looks deterministic and safe."}]},{"text":[{"text":"The model’s generation config sets "},{"text":"do_sample","code":true},{"text":" to false, so decoding is greedy and the temperature is ignored."}],"correct":true,"tempting":null},{"text":[{"text":"Temperature above 1.0 is clamped back to 1.0 by the library, which makes the distribution effectively deterministic for confident models."}],"correct":false,"tempting":[{"text":"Invents a clamp; values above 1 flatten the distribution and increase variety rather than removing it."}]},{"text":[{"text":"Sampling only applies once "},{"text":"top_p","code":true},{"text":" or "},{"text":"top_k","code":true},{"text":" is also set, because temperature alone has no candidate set to sample from."}],"correct":false,"tempting":[{"text":"Assumes a dependency between parameters; temperature alone still defines a full distribution to sample from."}]}],"order":[3,0,1,2],"why":null},"prev":{"slug":"padding-and-attention-masks","title":"Padding on the wrong side ruins generation quietly","minutes":9},"next":{"slug":"chat-templates","title":"The chat template is not optional","minutes":9}}}