{"course":{"slug":"running-models-yourself","title":"Running Models Yourself","url":"https://addaly.com/learn/running-models-yourself","level":"intermediate","track":"deep"},"lesson":{"slug":"speculative-decoding","title":"Speculative decoding: several tokens per weight read","url":"https://addaly.com/learn/running-models-yourself/speculative-decoding","position":41,"of":82,"minutes":9,"answer":"A draft model proposes several tokens and the full model verifies them in one pass with identical output distribution; the gain is entirely governed by acceptance rate.","keyIdea":"A draft model proposes several tokens and the full model verifies them in one pass with identical output distribution; the gain is entirely governed by acceptance rate, and it disappears on a heavily batched server where there is no idle compute left."},"body":{"format":"markdown","text":"## Exploiting the fact that decode wastes the machine\n\nGenerating one token reads every weight in the model and performs almost no arithmetic per byte. The hardware is idle in the sense that matters: it could have done far more work in the same memory traffic.\n\nSpeculative decoding uses that headroom. A cheap **draft** model proposes the next several tokens. The full model then evaluates all of them **in a single forward pass**, which costs almost exactly what one token would have cost, because it is the same weight read, and checks each proposal against what it would have produced. Accepted tokens are kept; at the first rejection the target model's own token is used and the rest are discarded.\n\nThe crucial property: the output distribution is provably identical to sampling from the target model alone. This is not an approximation and it does not trade quality for speed. That is unusual enough to be worth stating twice.\n\n## The arithmetic of when it helps\n\nLet the draft propose `k` tokens with acceptance rate `a`. Expected tokens per target-model pass is roughly `(1 - a^(k+1)) / (1 - a)`.\n\nAt `k = 4` and `a = 0.8`, that is about 3.4 tokens per pass. Subtract the draft's own cost (if the draft is a tenth the size, it adds roughly 40% of one target pass across four steps) and a realistic net speed-up is 2x to 2.5x.\n\n> Figure: Speculative decoding lives or dies on acceptance rate. Line chart of Multiplier against Acceptance rate, per cent: Tokens accepted per target pass, k = 4 (30, 1.4) (40, 1.7) (50, 1.9) (60, 2.3) (70, 2.8) (80, 3.4) (90, 4.1); Net speed-up after the draft's cost (30, 0.9) (40, 1.1) (50, 1.3) (60, 1.5) (70, 1.8) (80, 2.2) (90, 2.7). Below about 0.6 acceptance the draft model costs more than it saves. Code and structured output accept above 0.9 because most tokens are forced by syntax, and free prose accepts worst. The output distribution is provably identical either way, which makes this one of the very few speed gains with no quality trade at all.\n\nAt `a = 0.5`, expected tokens fall to about 1.9, and after the draft's cost the gain is slight. Below roughly 0.6 acceptance, speculation is not worth the complexity.\n\nSo the whole technique lives or dies on acceptance rate, which is a property of how well the draft imitates the target on **your** text.\n\n## What raises acceptance\n\n**Same family, same tokeniser.** A 0.5B and an 8B from the same release share training data and vocabulary and typically accept at 0.7 to 0.85. Two models from different families accept poorly and may not share a vocabulary at all, which makes speculation impossible rather than merely slow.\n\n**Predictable text.** Code, structured output, boilerplate and formatting accept extremely well (often above 0.9), because most tokens are forced by syntax. Free-form creative prose accepts worst. This is why speculative decoding is reported as transformative by people generating code and as marginal by people writing essays, and both reports are correct.\n\n**Low temperature.** Greedy decoding accepts far more than temperature 1.2, because the target's own choice is more predictable.\n\n## Draft-free variants\n\nA separate draft model costs memory you may not have. Two alternatives:\n\n**N-gram or prompt lookup.** Search the prompt and recent output for a repetition of the last few tokens, and propose whatever followed last time. Costs nothing, no extra weights, and works remarkably well when the output quotes the input: summarising, editing a document, rewriting code with small changes. Useless on novel text.\n\n**Self-speculation.** Run a subset of the model's own layers as the draft. No extra model to store, moderate acceptance.\n\n**EAGLE and similar trained heads.** A small head trained on the target model's own hidden states to predict its next tokens. Highest acceptance of the practical methods, needs a trained head for your specific model, and support is engine-dependent.\n\n## Turning it on\n\n```bash\n# llama.cpp: draft model plus target\nllama-server -m Qwen3-8B-Q4_K_M.gguf \\\n  -md Qwen3-0.6B-Q4_K_M.gguf --draft-max 5 --draft-min 1 -ngl 99\n```\n\n```bash\n# vLLM: n-gram speculation, no draft model\nvllm serve <model> --speculative-config \\\n  '{\"method\": \"ngram\", \"num_speculative_tokens\": 5, \"prompt_lookup_max\": 4}'\n```\n\nMeasure before and after on your own prompts. The gain varies by a factor of three across workloads, and the published figure was measured on somebody else's.\n\n## The honest catch\n\nSpeculation helps **single-stream latency** and does little for **aggregate throughput** on a busy server. The reason is direct: it exploits idle compute during memory-bound decoding, and batching has already claimed that idle compute for other users. At batch 64 the machine is compute-bound and there is no headroom left to speculate into. Worse, the rejected tokens are wasted compute you did pay for.\n\nSo: excellent for one person wanting a fast assistant, or for a latency-sensitive single-user service. Not the answer for a server with many concurrent requests, where the same effort spent on batching and prefix caching pays far better."},"check":{"question":"A team adds speculative decoding to a busy vLLM server running at batch 48 and measures a small drop in aggregate throughput, despite the same configuration doubling speed on a single-user test machine. Why?","options":[{"text":"The draft model competes for VRAM, reducing the cache blocks available and lowering the achievable batch size.","correct":false,"tempting":"A real secondary cost that would show up as reduced concurrency, not as slower throughput at the same batch size."},{"text":"Acceptance rate falls at high concurrency because the target model's distribution is affected by the other sequences in the batch.","correct":false,"tempting":"Invents an interaction between sequences; each sequence's logits are computed independently within a batch."},{"text":"Speculation converts idle compute into extra tokens, and a batched server has already used that compute for other users, so rejected drafts are pure waste.","correct":true},{"text":"Verification requires a separate forward pass per sequence, which serialises the batch and prevents continuous batching from re-forming.","correct":false,"tempting":"Misdescribes the mechanism; verification is one pass over the proposed tokens and integrates with the batch."}]},"prev":{"slug":"prefix-caching-properly","title":"Prefix caching, and how to design prompts for it","url":"https://addaly.com/learn/running-models-yourself/prefix-caching-properly"},"next":{"slug":"attention-kernels","title":"The kernels under everything: flash and paged attention","url":"https://addaly.com/learn/running-models-yourself/attention-kernels"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Speculative decoding: several tokens per weight read', in Running Models Yourself. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/running-models-yourself/speculative-decoding (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Speculative decoding: several tokens per weight read","courseTitle":"Running Models Yourself","module":{"index":5,"slug":"serving-and-performance","title":"Serving engines and performance"},"answer":null,"keyIdea":"A draft model proposes several tokens and the full model verifies them in one pass with identical output distribution; the gain is entirely governed by acceptance rate, and it disappears on a heavily batched server where there is no idle compute left.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"type":"line","title":"Speculative decoding lives or dies on acceptance rate","series":[{"label":"Tokens accepted per target pass, k = 4","points":[[30,1.4],[40,1.7],[50,1.9],[60,2.3],[70,2.8],[80,3.4],[90,4.1]]},{"label":"Net speed-up after the draft’s cost","points":[[30,0.9],[40,1.1],[50,1.3],[60,1.5],[70,1.8],[80,2.2],[90,2.7]]}],"xLabel":"Acceptance rate, per cent","yLabel":"Multiplier","caption":"Below about 0.6 acceptance the draft model costs more than it saves. Code and structured output accept above 0.9 because most tokens are forced by syntax, and free prose accepts worst. The output distribution is provably identical either way, which makes this one of the very few speed gains with no quality trade at all."},"alt":"Figure: Speculative decoding lives or dies on acceptance rate. Line chart of Multiplier against Acceptance rate, per cent: Tokens accepted per target pass, k = 4 (30, 1.4) (40, 1.7) (50, 1.9) (60, 2.3) (70, 2.8) (80, 3.4) (90, 4.1); Net speed-up after the draft’s cost (30, 0.9) (40, 1.1) (50, 1.3) (60, 1.5) (70, 1.8) (80, 2.2) (90, 2.7). Below about 0.6 acceptance the draft model costs more than it saves. Code and structured output accept above 0.9 because most tokens are forced by syntax, and free prose accepts worst. The output distribution is provably identical either way, which makes this one of the very few speed gains with no quality trade at all."}},"blocks":[{"t":"h2","id":"exploiting-the-fact-that-decode-wastes-the-machine","runs":[{"text":"Exploiting the fact that decode wastes the machine"}]},{"t":"p","runs":[{"text":"Generating one token reads every weight in the model and performs almost no arithmetic per byte. The hardware is idle in the sense that matters: it could have done far more work in the same memory traffic."}]},{"t":"p","runs":[{"text":"Speculative decoding uses that headroom. A cheap "},{"text":"draft","b":true},{"text":" model proposes the next several tokens. The full model then evaluates all of them "},{"text":"in a single forward pass","b":true},{"text":", which costs almost exactly what one token would have cost, because it is the same weight read, and checks each proposal against what it would have produced. Accepted tokens are kept; at the first rejection the target model’s own token is used and the rest are discarded."}]},{"t":"p","runs":[{"text":"The crucial property: the output distribution is provably identical to sampling from the target model alone. This is not an approximation and it does not trade quality for speed. That is unusual enough to be worth stating twice."}]},{"t":"h2","id":"the-arithmetic-of-when-it-helps","runs":[{"text":"The arithmetic of when it helps"}]},{"t":"p","runs":[{"text":"Let the draft propose "},{"text":"k","code":true},{"text":" tokens with acceptance rate "},{"text":"a","code":true},{"text":". Expected tokens per target-model pass is roughly "},{"text":"(1 - a^(k+1)) / (1 - a)","code":true},{"text":"."}]},{"t":"p","runs":[{"text":"At "},{"text":"k = 4","code":true},{"text":" and "},{"text":"a = 0.8","code":true},{"text":", that is about 3.4 tokens per pass. Subtract the draft’s own cost (if the draft is a tenth the size, it adds roughly 40% of one target pass across four steps) and a realistic net speed-up is 2x to 2.5x."}]},{"t":"p","runs":[{"text":"At "},{"text":"a = 0.5","code":true},{"text":", expected tokens fall to about 1.9, and after the draft’s cost the gain is slight. Below roughly 0.6 acceptance, speculation is not worth the complexity."}]},{"t":"p","runs":[{"text":"So the whole technique lives or dies on acceptance rate, which is a property of how well the draft imitates the target on "},{"text":"your","b":true},{"text":" text."}]},{"t":"h2","id":"what-raises-acceptance","runs":[{"text":"What raises acceptance"}]},{"t":"p","runs":[{"text":"Same family, same tokeniser.","b":true},{"text":" A 0.5B and an 8B from the same release share training data and vocabulary and typically accept at 0.7 to 0.85. Two models from different families accept poorly and may not share a vocabulary at all, which makes speculation impossible rather than merely slow."}]},{"t":"p","runs":[{"text":"Predictable text.","b":true},{"text":" Code, structured output, boilerplate and formatting accept extremely well (often above 0.9), because most tokens are forced by syntax. Free-form creative prose accepts worst. This is why speculative decoding is reported as transformative by people generating code and as marginal by people writing essays, and both reports are correct."}]},{"t":"p","runs":[{"text":"Low temperature.","b":true},{"text":" Greedy decoding accepts far more than temperature 1.2, because the target’s own choice is more predictable."}]},{"t":"h2","id":"draft-free-variants","runs":[{"text":"Draft-free variants"}]},{"t":"p","runs":[{"text":"A separate draft model costs memory you may not have. Two alternatives:"}]},{"t":"p","runs":[{"text":"N-gram or prompt lookup.","b":true},{"text":" Search the prompt and recent output for a repetition of the last few tokens, and propose whatever followed last time. Costs nothing, no extra weights, and works remarkably well when the output quotes the input: summarising, editing a document, rewriting code with small changes. Useless on novel text."}]},{"t":"p","runs":[{"text":"Self-speculation.","b":true},{"text":" Run a subset of the model’s own layers as the draft. No extra model to store, moderate acceptance."}]},{"t":"p","runs":[{"text":"EAGLE and similar trained heads.","b":true},{"text":" A small head trained on the target model’s own hidden states to predict its next tokens. Highest acceptance of the practical methods, needs a trained head for your specific model, and support is engine-dependent."}]},{"t":"h2","id":"turning-it-on","runs":[{"text":"Turning it on"}]},{"t":"code","lang":"bash","label":"Bash","text":"# llama.cpp: draft model plus target\nllama-server -m Qwen3-8B-Q4_K_M.gguf \\\n  -md Qwen3-0.6B-Q4_K_M.gguf --draft-max 5 --draft-min 1 -ngl 99"},{"t":"code","lang":"bash","label":"Bash","text":"# vLLM: n-gram speculation, no draft model\nvllm serve <model> --speculative-config \\\n  '{\"method\": \"ngram\", \"num_speculative_tokens\": 5, \"prompt_lookup_max\": 4}'"},{"t":"p","runs":[{"text":"Measure before and after on your own prompts. The gain varies by a factor of three across workloads, and the published figure was measured on somebody else’s."}]},{"t":"h2","id":"the-honest-catch","runs":[{"text":"The honest catch"}]},{"t":"p","runs":[{"text":"Speculation helps "},{"text":"single-stream latency","b":true},{"text":" and does little for "},{"text":"aggregate throughput","b":true},{"text":" on a busy server. The reason is direct: it exploits idle compute during memory-bound decoding, and batching has already claimed that idle compute for other users. At batch 64 the machine is compute-bound and there is no headroom left to speculate into. Worse, the rejected tokens are wasted compute you did pay for."}]},{"t":"p","runs":[{"text":"So: excellent for one person wanting a fast assistant, or for a latency-sensitive single-user service. Not the answer for a server with many concurrent requests, where the same effort spent on batching and prefix caching pays far better."}]}],"check":{"question":[{"text":"A team adds speculative decoding to a busy vLLM server running at batch 48 and measures a small drop in aggregate throughput, despite the same configuration doubling speed on a single-user test machine. Why?"}],"options":[{"text":[{"text":"The draft model competes for VRAM, reducing the cache blocks available and lowering the achievable batch size."}],"correct":false,"tempting":[{"text":"A real secondary cost that would show up as reduced concurrency, not as slower throughput at the same batch size."}]},{"text":[{"text":"Acceptance rate falls at high concurrency because the target model’s distribution is affected by the other sequences in the batch."}],"correct":false,"tempting":[{"text":"Invents an interaction between sequences; each sequence’s logits are computed independently within a batch."}]},{"text":[{"text":"Speculation converts idle compute into extra tokens, and a batched server has already used that compute for other users, so rejected drafts are pure waste."}],"correct":true,"tempting":null},{"text":[{"text":"Verification requires a separate forward pass per sequence, which serialises the batch and prevents continuous batching from re-forming."}],"correct":false,"tempting":[{"text":"Misdescribes the mechanism; verification is one pass over the proposed tokens and integrates with the batch."}]}],"order":[3,0,2,1],"why":null},"prev":{"slug":"prefix-caching-properly","title":"Prefix caching, and how to design prompts for it","minutes":9},"next":{"slug":"attention-kernels","title":"The kernels under everything: flash and paged attention","minutes":9}}}