{"course":{"slug":"how-llms-work","title":"How a Language Model Actually Works","url":"https://addaly.com/learn/how-llms-work","level":"intermediate","track":"start"},"lesson":{"slug":"greedy-beam-and-degeneration","title":"Greedy, beam search, and why chat models refuse both","url":"https://addaly.com/learn/how-llms-work/greedy-beam-and-degeneration","position":21,"of":78,"minutes":9,"answer":"Always taking the most likely token produces text that is more probable and less human, which is why generation samples instead of maximising.","keyIdea":"Always taking the most likely token produces text that is more probable and less human, which is why generation samples instead of maximising."},"body":{"format":"markdown","text":"## The obvious strategy, and what it does\n\nYou have a distribution over 128,256 tokens. The obvious move is to take the most likely one, append it, and repeat. That is greedy decoding, and it is two lines:\n\n```python\nfor _ in range(max_tokens):\n    logits = model(ids)[-1]\n    ids.append(int(logits.argmax()))\n```\n\nTry it on an open model with an open-ended prompt. Within fifty tokens you will usually see something like this:\n\n```text\nThe best way to learn a language is to practise every day. The best way to\nlearn a language is to practise every day. The best way to learn a language...\n```\n\nIt locks into a loop. This is not a bug in any particular model; it is a documented and general property, described carefully in work on neural text degeneration in 2019, and it happens across model families and sizes.\n\n## Why maximising probability produces bad text\n\nThe finding that made this click is counter-intuitive. Human-written text does **not** consist of high-probability tokens. Measure the per-token probability of real writing under a language model and you see constant variation: probable words, then a surprising one, then probable again. Greedy decoding produces a flat, uniformly high-probability sequence, which is a shape that human text never has.\n\n> Figure: Two ways to turn a distribution into a sentence. Maximise: greedy and beam search: Takes the most probable continuation at every step; Produces flat, uniformly high-probability text; Locks into loops, helped along by induction; Right for translation, transcription, extraction, code that must compile. Sample: what chat models do: Draws from the distribution, with the pool trimmed by top-k or top-p; Restores the up-and-down that real writing has; Escapes loops, because a less likely token can occasionally win; Right for anything open-ended. There is no setting that gives you human-shaped text and predictable text at the same time. Choose by task, and buy consistency by evaluating many runs rather than by hunting for a decoding flag.\n\nThere is a second, self-reinforcing effect. Once \"The best way to learn a language is to practise every day.\" appears in the context, the induction mechanism from module two makes repeating it the single most likely continuation. Each repetition raises the probability of the next. Greedy decoding walks straight into that trap and cannot get out, because it has no mechanism for taking a less likely path.\n\n## Beam search: better maximising, same problem\n\nBeam search keeps the *k* best partial sequences at each step rather than one, scoring whole sequences instead of single tokens. It is better at finding a high-probability sequence, and it is the right tool for translation and speech recognition, where there is a single correct output and the model's probability is a good proxy for correctness.\n\nFor open-ended generation it makes things worse in a specific way: it finds *even more* probable text, and more probable means blander and more repetitive. It also costs *k* times the compute, and it interacts badly with the key-value cache, since each beam needs its own. This is why beam search, once standard in machine translation, is essentially absent from chat model serving.\n\n## Penalties, and their side effects\n\nThe blunt instruments for repetition:\n\n- **Repetition penalty** divides the logit of any token already present, discouraging reuse regardless of how far back it appeared.\n- **Frequency penalty** subtracts an amount proportional to how many times a token has occurred.\n- **Presence penalty** subtracts a fixed amount if a token has occurred at all.\n- **No-repeat n-gram blocking** forbids any n-gram from appearing twice: effective and dangerous.\n\nEvery one of these is content-blind, and that is the honest caveat. A penalty that stops \"the best way\" repeating also penalises the second occurrence of a variable name in code, the recurring subject of a paragraph, and the word \"patient\" in a medical summary. Blocking repeated 4-grams will break a citation format. The usual practical advice is to keep these low, 1.0 to 1.15 for repetition penalty, and prefer better sampling to heavy penalties.\n\n## What is actually used\n\nChat models sample: draw from the distribution rather than maximising it, then restrict the pool with top-k, top-p or min-p, which the next lesson covers in detail. Sampling reintroduces the variation that real text has, and it escapes loops because a lower-probability token can win occasionally.\n\nThe trade-off is stated plainly and it never goes away. Sampling makes output more human and less predictable. Maximising makes it more predictable and less human. There is no setting that gives you both, and any advice that promises one is describing a preference rather than a mechanism.\n\n## Choosing, by task\n\n- **Extraction, classification, code that must compile, any task with one right answer**: greedy or near-greedy. Low temperature, small pool. The failure mode of blandness does not apply when there is nothing to be creative about.\n- **Translation and transcription**: beam search remains reasonable, and the tooling for it exists.\n- **Anything open-ended**: sample. Accept the variation, and if you need consistency, get it from evaluation over many runs rather than from a decoding setting.\n\n## The test worth running\n\nTake one prompt. Generate ten completions greedily and ten with sampling at temperature 0.8. Read all twenty. The greedy ten will be near-identical to each other and duller than you expected; the sampled ten will vary in quality in both directions. That five-minute exercise settles the question better than any argument about settings."},"check":{"question":"A summarisation tool switched from sampling to greedy decoding to make output more consistent, and now some summaries repeat a sentence several times. What is the mechanism?","options":[{"text":"Greedy decoding removes the repetition penalty, which was previously suppressing the loop.","correct":false,"tempting":"Penalties and decoding strategy are configured together in most libraries, so assuming one disabled the other is an easy slip."},{"text":"Once a phrase appears in the context it becomes the most likely continuation, and greedy decoding has no way to take a less likely path out of that loop.","correct":true},{"text":"The model's context window filled up, causing earlier text to be re-processed.","correct":false,"tempting":"Context overflow does cause strange behaviour, and repetition is a plausible symptom of a truncation bug."},{"text":"Greedy decoding disables the end-of-sequence token, so generation continues past the natural stop.","correct":false,"tempting":"Runaway generation is a real symptom of stop-token misconfiguration, which makes this a sensible operational suspicion."}]},"prev":{"slug":"next-token-prediction","title":"Next-token prediction, and where the reasoning comes from","url":"https://addaly.com/learn/how-llms-work/next-token-prediction"},"next":{"slug":"temperature-and-sampling","title":"Temperature and sampling: what the knob really does","url":"https://addaly.com/learn/how-llms-work/temperature-and-sampling"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Greedy, beam search, and why chat models refuse both', in How a Language Model Actually Works. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/how-llms-work/greedy-beam-and-degeneration (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Greedy, beam search, and why chat models refuse both","courseTitle":"How a Language Model Actually Works","module":{"index":3,"slug":"turning-scores-into-text","title":"Turning scores into text"},"answer":null,"keyIdea":"Always taking the most likely token produces text that is more probable and less human, which is why generation samples instead of maximising.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"left":{"title":"Maximise: greedy and beam search","items":["Takes the most probable continuation at every step","Produces flat, uniformly high-probability text","Locks into loops, helped along by induction","Right for translation, transcription, extraction, code that must compile"]},"type":"compare","right":{"title":"Sample: what chat models do","items":["Draws from the distribution, with the pool trimmed by top-k or top-p","Restores the up-and-down that real writing has","Escapes loops, because a less likely token can occasionally win","Right for anything open-ended"]},"title":"Two ways to turn a distribution into a sentence","caption":"There is no setting that gives you human-shaped text and predictable text at the same time. Choose by task, and buy consistency by evaluating many runs rather than by hunting for a decoding flag."},"alt":"Figure: Two ways to turn a distribution into a sentence. Maximise: greedy and beam search: Takes the most probable continuation at every step; Produces flat, uniformly high-probability text; Locks into loops, helped along by induction; Right for translation, transcription, extraction, code that must compile. Sample: what chat models do: Draws from the distribution, with the pool trimmed by top-k or top-p; Restores the up-and-down that real writing has; Escapes loops, because a less likely token can occasionally win; Right for anything open-ended. There is no setting that gives you human-shaped text and predictable text at the same time. Choose by task, and buy consistency by evaluating many runs rather than by hunting for a decoding flag."}},"blocks":[{"t":"h2","id":"the-obvious-strategy-and-what-it-does","runs":[{"text":"The obvious strategy, and what it does"}]},{"t":"p","runs":[{"text":"You have a distribution over 128,256 tokens. The obvious move is to take the most likely one, append it, and repeat. That is greedy decoding, and it is two lines:"}]},{"t":"code","lang":"python","label":"Python","text":"for _ in range(max_tokens):\n    logits = model(ids)[-1]\n    ids.append(int(logits.argmax()))"},{"t":"p","runs":[{"text":"Try it on an open model with an open-ended prompt. Within fifty tokens you will usually see something like this:"}]},{"t":"code","lang":"text","label":"Text","text":"The best way to learn a language is to practise every day. The best way to\nlearn a language is to practise every day. The best way to learn a language..."},{"t":"p","runs":[{"text":"It locks into a loop. This is not a bug in any particular model; it is a documented and general property, described carefully in work on neural text degeneration in 2019, and it happens across model families and sizes."}]},{"t":"h2","id":"why-maximising-probability-produces-bad-text","runs":[{"text":"Why maximising probability produces bad text"}]},{"t":"p","runs":[{"text":"The finding that made this click is counter-intuitive. Human-written text does "},{"text":"not","b":true},{"text":" consist of high-probability tokens. Measure the per-token probability of real writing under a language model and you see constant variation: probable words, then a surprising one, then probable again. Greedy decoding produces a flat, uniformly high-probability sequence, which is a shape that human text never has."}]},{"t":"p","runs":[{"text":"There is a second, self-reinforcing effect. Once “The best way to learn a language is to practise every day.” appears in the context, the induction mechanism from module two makes repeating it the single most likely continuation. Each repetition raises the probability of the next. Greedy decoding walks straight into that trap and cannot get out, because it has no mechanism for taking a less likely path."}]},{"t":"h2","id":"beam-search-better-maximising-same-problem","runs":[{"text":"Beam search: better maximising, same problem"}]},{"t":"p","runs":[{"text":"Beam search keeps the "},{"text":"k","i":true},{"text":" best partial sequences at each step rather than one, scoring whole sequences instead of single tokens. It is better at finding a high-probability sequence, and it is the right tool for translation and speech recognition, where there is a single correct output and the model’s probability is a good proxy for correctness."}]},{"t":"p","runs":[{"text":"For open-ended generation it makes things worse in a specific way: it finds "},{"text":"even more","i":true},{"text":" probable text, and more probable means blander and more repetitive. It also costs "},{"text":"k","i":true},{"text":" times the compute, and it interacts badly with the key-value cache, since each beam needs its own. This is why beam search, once standard in machine translation, is essentially absent from chat model serving."}]},{"t":"h2","id":"penalties-and-their-side-effects","runs":[{"text":"Penalties, and their side effects"}]},{"t":"p","runs":[{"text":"The blunt instruments for repetition:"}]},{"t":"ul","items":[[{"text":"Repetition penalty","b":true},{"text":" divides the logit of any token already present, discouraging reuse regardless of how far back it appeared."}],[{"text":"Frequency penalty","b":true},{"text":" subtracts an amount proportional to how many times a token has occurred."}],[{"text":"Presence penalty","b":true},{"text":" subtracts a fixed amount if a token has occurred at all."}],[{"text":"No-repeat n-gram blocking","b":true},{"text":" forbids any n-gram from appearing twice: effective and dangerous."}]]},{"t":"p","runs":[{"text":"Every one of these is content-blind, and that is the honest caveat. A penalty that stops “the best way” repeating also penalises the second occurrence of a variable name in code, the recurring subject of a paragraph, and the word “patient” in a medical summary. Blocking repeated 4-grams will break a citation format. The usual practical advice is to keep these low, 1.0 to 1.15 for repetition penalty, and prefer better sampling to heavy penalties."}]},{"t":"h2","id":"what-is-actually-used","runs":[{"text":"What is actually used"}]},{"t":"p","runs":[{"text":"Chat models sample: draw from the distribution rather than maximising it, then restrict the pool with top-k, top-p or min-p, which the next lesson covers in detail. Sampling reintroduces the variation that real text has, and it escapes loops because a lower-probability token can win occasionally."}]},{"t":"p","runs":[{"text":"The trade-off is stated plainly and it never goes away. Sampling makes output more human and less predictable. Maximising makes it more predictable and less human. There is no setting that gives you both, and any advice that promises one is describing a preference rather than a mechanism."}]},{"t":"h2","id":"choosing-by-task","runs":[{"text":"Choosing, by task"}]},{"t":"ul","items":[[{"text":"Extraction, classification, code that must compile, any task with one right answer","b":true},{"text":": greedy or near-greedy. Low temperature, small pool. The failure mode of blandness does not apply when there is nothing to be creative about."}],[{"text":"Translation and transcription","b":true},{"text":": beam search remains reasonable, and the tooling for it exists."}],[{"text":"Anything open-ended","b":true},{"text":": sample. Accept the variation, and if you need consistency, get it from evaluation over many runs rather than from a decoding setting."}]]},{"t":"h2","id":"the-test-worth-running","runs":[{"text":"The test worth running"}]},{"t":"p","runs":[{"text":"Take one prompt. Generate ten completions greedily and ten with sampling at temperature 0.8. Read all twenty. The greedy ten will be near-identical to each other and duller than you expected; the sampled ten will vary in quality in both directions. That five-minute exercise settles the question better than any argument about settings."}]}],"check":{"question":[{"text":"A summarisation tool switched from sampling to greedy decoding to make output more consistent, and now some summaries repeat a sentence several times. What is the mechanism?"}],"options":[{"text":[{"text":"Greedy decoding removes the repetition penalty, which was previously suppressing the loop."}],"correct":false,"tempting":[{"text":"Penalties and decoding strategy are configured together in most libraries, so assuming one disabled the other is an easy slip."}]},{"text":[{"text":"Once a phrase appears in the context it becomes the most likely continuation, and greedy decoding has no way to take a less likely path out of that loop."}],"correct":true,"tempting":null},{"text":[{"text":"The model’s context window filled up, causing earlier text to be re-processed."}],"correct":false,"tempting":[{"text":"Context overflow does cause strange behaviour, and repetition is a plausible symptom of a truncation bug."}]},{"text":[{"text":"Greedy decoding disables the end-of-sequence token, so generation continues past the natural stop."}],"correct":false,"tempting":[{"text":"Runaway generation is a real symptom of stop-token misconfiguration, which makes this a sensible operational suspicion."}]}],"order":[0,1,2,3],"why":null},"prev":{"slug":"next-token-prediction","title":"Next-token prediction, and where the reasoning comes from","minutes":8},"next":{"slug":"temperature-and-sampling","title":"Temperature and sampling: what the knob really does","minutes":6}}}