{"course":{"slug":"how-llms-work","title":"How a Language Model Actually Works","url":"https://addaly.com/learn/how-llms-work","level":"intermediate","track":"start"},"lesson":{"slug":"speculative-decoding","title":"Speculative decoding: a small model guesses, a big model checks","url":"https://addaly.com/learn/how-llms-work/speculative-decoding","position":27,"of":78,"minutes":8,"answer":"A large model can verify several drafted tokens in one pass for almost the cost of producing one, so a small draft model makes generation faster with no change to the output distribution.","keyIdea":"A large model can verify several drafted tokens in one pass for almost the cost of producing one, so a small draft model makes generation faster with no change to the output distribution."},"body":{"format":"markdown","text":"## The idle machine\n\nDecoding is memory-bound. Each step reads every weight of the model to produce a single token, and the arithmetic units spend most of their time waiting. That means checking *five* tokens costs barely more than checking one: the weights are read once either way.\n\nSpeculative decoding exploits this directly, and it is one of the few optimisations that is free of trade-offs.\n\n## The algorithm\n\n1. A small, fast **draft model** generates the next *k* tokens, say four. This is cheap: a 1B draft model is roughly a tenth of the cost of a 70B target model per token.\n2. The large **target model** processes all four in a single forward pass, producing its own distribution at each of those positions, which it can do because the causal mask means each position's output depends only on what precedes it.\n3. Compare. Accept the drafted tokens as long as the target model agrees, by a rule that provably preserves the target's distribution. At the first disagreement, discard the rest and take the target's own token.\n4. Repeat.\n\nIf all four are accepted you have produced five tokens for the cost of one target pass plus four cheap draft passes. If none are, you have produced one and wasted a little.\n\n> Figure: One round of speculative decoding. Steps: 1. Draft four tokens with a small model (Roughly a tenth of the target model's cost per token); 2. Check all four in one target pass (The causal mask means each position's output depends only on what precedes it); 3. Accept by the rejection-sampling rule (Not \"does the top choice match\": a rule that preserves the target's own distribution); 4. Stop at the first disagreement (Take the target's token there and throw the rest of the draft away); 5. Five tokens for one target pass, or one and a little waste (Gains of two to three times, set entirely by the acceptance rate). Decode is memory-bound, so checking five tokens costs barely more than checking one: the weights are read once either way. The output distribution is provably identical to sampling from the target alone, which is why this is a speed change and not a quality trade.\n\n## The guarantee that makes it acceptable\n\nThe acceptance rule is the clever part. It is not \"accept if the target's top choice matches\". It is a rejection-sampling procedure that accepts or rejects each drafted token with a probability derived from both models' distributions, and on rejection samples from an adjusted distribution.\n\nThe result, proved in the original work, is that the output distribution is **identical** to sampling from the target model alone. This is not an approximation and it is not a quality trade. Anyone telling you speculative decoding makes output slightly worse has misunderstood it or is describing a different technique.\n\n## Why speed-ups are what they are\n\nTypical reported gains are two to three times. The limit is the acceptance rate: how often the small model guesses what the big one would have said.\n\nAcceptance is high on predictable text: boilerplate, code with obvious continuations, the second half of common phrases, structured formats. It is low exactly where the target model is doing something the draft model cannot: unusual reasoning, rare facts, creative phrasing.\n\nSo the speed-up varies by content, which is worth knowing when you benchmark: measuring on repetitive text will flatter the technique, and measuring on hard reasoning will understate it.\n\n## Variants you will meet\n\n**Self-speculation.** Use the target model's own early layers, or a few extra prediction heads bolted on, as the draft. No second model to host. Medusa and EAGLE are the well-known families here.\n\n**N-gram or prompt lookup.** For summarisation, editing and retrieval-augmented answers, much of the output is copied from the input. Draft by looking up the next few tokens from the prompt where the recent text matches. There is no draft model at all, it is nearly free, and on copy-heavy tasks it works surprisingly well.\n\n**Multi-token prediction.** Train the model from the start to predict several future tokens at once, giving it a built-in draft. Some recent large models include this.\n\n## What it means for you\n\nIf you use a hosted API, this is likely already happening and it is part of why per-token latency improved without model changes. You cannot control it and you do not need to.\n\nIf you self-host, it is one of the cheaper wins available. vLLM and `llama.cpp` both support speculative decoding, both are free, and the requirement is a small model sharing the target's tokenizer: usually the smallest member of the same model family.\n\nThe deeper point is the one to keep. A large part of what looks like models getting faster is not better models at all. It is serving engineering (batching, caching, paged memory, speculation) squeezing an inefficient workload. Knowing which is which stops you from attributing an infrastructure improvement to a capability improvement."},"check":{"question":"Speculative decoding gives a 2.5× speed-up on a code assistant but only 1.3× on an open-ended writing tool. What explains the difference?","options":[{"text":"The writing tool uses a longer context, and speculation degrades with context length.","correct":false,"tempting":"Context length affects almost everything in serving, so tying a slowdown to it is usually a safe bet."},{"text":"Creative writing uses a higher temperature, and speculation only applies at temperature zero.","correct":false,"tempting":"Temperature does interact with the acceptance rule, so restricting speculation to greedy decoding sounds like a reasonable limitation."},{"text":"Code contains far more predictable continuations, so the draft model's guesses are accepted more often, and the speed-up is governed by the acceptance rate.","correct":true},{"text":"The code model's outputs are shorter, so the fixed overhead is amortised better.","correct":false,"tempting":"Fixed overheads really do dominate short generations, which makes this a plausible piece of performance reasoning."}]},"prev":{"slug":"sampling-many-answers","title":"Asking the same question several times","url":"https://addaly.com/learn/how-llms-work/sampling-many-answers"},"next":{"slug":"temperature-zero-is-not-deterministic","title":"Why temperature zero still gives you different answers","url":"https://addaly.com/learn/how-llms-work/temperature-zero-is-not-deterministic"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Speculative decoding: a small model guesses, a big model checks', in How a Language Model Actually Works. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/how-llms-work/speculative-decoding (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Speculative decoding: a small model guesses, a big model checks","courseTitle":"How a Language Model Actually Works","module":{"index":3,"slug":"turning-scores-into-text","title":"Turning scores into text"},"answer":null,"keyIdea":"A large model can verify several drafted tokens in one pass for almost the cost of producing one, so a small draft model makes generation faster with no change to the output distribution.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"type":"flow","steps":[{"label":"Draft four tokens with a small model","detail":"Roughly a tenth of the target model’s cost per token"},{"label":"Check all four in one target pass","detail":"The causal mask means each position’s output depends only on what precedes it"},{"label":"Accept by the rejection-sampling rule","detail":"Not “does the top choice match”: a rule that preserves the target’s own distribution"},{"label":"Stop at the first disagreement","detail":"Take the target’s token there and throw the rest of the draft away"},{"label":"Five tokens for one target pass, or one and a little waste","detail":"Gains of two to three times, set entirely by the acceptance rate"}],"title":"One round of speculative decoding","caption":"Decode is memory-bound, so checking five tokens costs barely more than checking one: the weights are read once either way. The output distribution is provably identical to sampling from the target alone, which is why this is a speed change and not a quality trade."},"alt":"Figure: One round of speculative decoding. Steps: 1. Draft four tokens with a small model (Roughly a tenth of the target model’s cost per token); 2. Check all four in one target pass (The causal mask means each position’s output depends only on what precedes it); 3. Accept by the rejection-sampling rule (Not “does the top choice match”: a rule that preserves the target’s own distribution); 4. Stop at the first disagreement (Take the target’s token there and throw the rest of the draft away); 5. Five tokens for one target pass, or one and a little waste (Gains of two to three times, set entirely by the acceptance rate). Decode is memory-bound, so checking five tokens costs barely more than checking one: the weights are read once either way. The output distribution is provably identical to sampling from the target alone, which is why this is a speed change and not a quality trade."}},"blocks":[{"t":"h2","id":"the-idle-machine","runs":[{"text":"The idle machine"}]},{"t":"p","runs":[{"text":"Decoding is memory-bound. Each step reads every weight of the model to produce a single token, and the arithmetic units spend most of their time waiting. That means checking "},{"text":"five","i":true},{"text":" tokens costs barely more than checking one: the weights are read once either way."}]},{"t":"p","runs":[{"text":"Speculative decoding exploits this directly, and it is one of the few optimisations that is free of trade-offs."}]},{"t":"h2","id":"the-algorithm","runs":[{"text":"The algorithm"}]},{"t":"ol","start":1,"items":[[{"text":"A small, fast "},{"text":"draft model","b":true},{"text":" generates the next "},{"text":"k","i":true},{"text":" tokens, say four. This is cheap: a 1B draft model is roughly a tenth of the cost of a 70B target model per token."}],[{"text":"The large "},{"text":"target model","b":true},{"text":" processes all four in a single forward pass, producing its own distribution at each of those positions, which it can do because the causal mask means each position’s output depends only on what precedes it."}],[{"text":"Compare. Accept the drafted tokens as long as the target model agrees, by a rule that provably preserves the target’s distribution. At the first disagreement, discard the rest and take the target’s own token."}],[{"text":"Repeat."}]]},{"t":"p","runs":[{"text":"If all four are accepted you have produced five tokens for the cost of one target pass plus four cheap draft passes. If none are, you have produced one and wasted a little."}]},{"t":"h2","id":"the-guarantee-that-makes-it-acceptable","runs":[{"text":"The guarantee that makes it acceptable"}]},{"t":"p","runs":[{"text":"The acceptance rule is the clever part. It is not “accept if the target’s top choice matches”. It is a rejection-sampling procedure that accepts or rejects each drafted token with a probability derived from both models’ distributions, and on rejection samples from an adjusted distribution."}]},{"t":"p","runs":[{"text":"The result, proved in the original work, is that the output distribution is "},{"text":"identical","b":true},{"text":" to sampling from the target model alone. This is not an approximation and it is not a quality trade. Anyone telling you speculative decoding makes output slightly worse has misunderstood it or is describing a different technique."}]},{"t":"h2","id":"why-speed-ups-are-what-they-are","runs":[{"text":"Why speed-ups are what they are"}]},{"t":"p","runs":[{"text":"Typical reported gains are two to three times. The limit is the acceptance rate: how often the small model guesses what the big one would have said."}]},{"t":"p","runs":[{"text":"Acceptance is high on predictable text: boilerplate, code with obvious continuations, the second half of common phrases, structured formats. It is low exactly where the target model is doing something the draft model cannot: unusual reasoning, rare facts, creative phrasing."}]},{"t":"p","runs":[{"text":"So the speed-up varies by content, which is worth knowing when you benchmark: measuring on repetitive text will flatter the technique, and measuring on hard reasoning will understate it."}]},{"t":"h2","id":"variants-you-will-meet","runs":[{"text":"Variants you will meet"}]},{"t":"p","runs":[{"text":"Self-speculation.","b":true},{"text":" Use the target model’s own early layers, or a few extra prediction heads bolted on, as the draft. No second model to host. Medusa and EAGLE are the well-known families here."}]},{"t":"p","runs":[{"text":"N-gram or prompt lookup.","b":true},{"text":" For summarisation, editing and retrieval-augmented answers, much of the output is copied from the input. Draft by looking up the next few tokens from the prompt where the recent text matches. There is no draft model at all, it is nearly free, and on copy-heavy tasks it works surprisingly well."}]},{"t":"p","runs":[{"text":"Multi-token prediction.","b":true},{"text":" Train the model from the start to predict several future tokens at once, giving it a built-in draft. Some recent large models include this."}]},{"t":"h2","id":"what-it-means-for-you","runs":[{"text":"What it means for you"}]},{"t":"p","runs":[{"text":"If you use a hosted API, this is likely already happening and it is part of why per-token latency improved without model changes. You cannot control it and you do not need to."}]},{"t":"p","runs":[{"text":"If you self-host, it is one of the cheaper wins available. vLLM and "},{"text":"llama.cpp","code":true},{"text":" both support speculative decoding, both are free, and the requirement is a small model sharing the target’s tokenizer: usually the smallest member of the same model family."}]},{"t":"p","runs":[{"text":"The deeper point is the one to keep. A large part of what looks like models getting faster is not better models at all. It is serving engineering (batching, caching, paged memory, speculation) squeezing an inefficient workload. Knowing which is which stops you from attributing an infrastructure improvement to a capability improvement."}]}],"check":{"question":[{"text":"Speculative decoding gives a 2.5× speed-up on a code assistant but only 1.3× on an open-ended writing tool. What explains the difference?"}],"options":[{"text":[{"text":"The writing tool uses a longer context, and speculation degrades with context length."}],"correct":false,"tempting":[{"text":"Context length affects almost everything in serving, so tying a slowdown to it is usually a safe bet."}]},{"text":[{"text":"Creative writing uses a higher temperature, and speculation only applies at temperature zero."}],"correct":false,"tempting":[{"text":"Temperature does interact with the acceptance rule, so restricting speculation to greedy decoding sounds like a reasonable limitation."}]},{"text":[{"text":"Code contains far more predictable continuations, so the draft model’s guesses are accepted more often, and the speed-up is governed by the acceptance rate."}],"correct":true,"tempting":null},{"text":[{"text":"The code model’s outputs are shorter, so the fixed overhead is amortised better."}],"correct":false,"tempting":[{"text":"Fixed overheads really do dominate short generations, which makes this a plausible piece of performance reasoning."}]}],"order":[1,0,3,2],"why":null},"prev":{"slug":"sampling-many-answers","title":"Asking the same question several times","minutes":8},"next":{"slug":"temperature-zero-is-not-deterministic","title":"Why temperature zero still gives you different answers","minutes":8}}}