Speculative decoding: several tokens per weight read

Exploiting the fact that decode wastes the machine

Generating one token reads every weight in the model and performs almost no arithmetic per byte. The hardware is idle in the sense that matters: it could have done far more work in the same memory traffic.

Speculative decoding uses that headroom. A cheap draft model proposes the next several tokens. The full model then evaluates all of them in a single forward pass, which costs almost exactly what one token would have cost, because it is the same weight read, and checks each proposal against what it would have produced. Accepted tokens are kept; at the first rejection the target model’s own token is used and the rest are discarded.

The crucial property: the output distribution is provably identical to sampling from the target model alone. This is not an approximation and it does not trade quality for speed. That is unusual enough to be worth stating twice.

The arithmetic of when it helps

Let the draft propose k tokens with acceptance rate a. Expected tokens per target-model pass is roughly (1 - a^(k+1)) / (1 - a).

At k = 4 and a = 0.8, that is about 3.4 tokens per pass. Subtract the draft’s own cost (if the draft is a tenth the size, it adds roughly 40% of one target pass across four steps) and a realistic net speed-up is 2x to 2.5x.

At a = 0.5, expected tokens fall to about 1.9, and after the draft’s cost the gain is slight. Below roughly 0.6 acceptance, speculation is not worth the complexity.

So the whole technique lives or dies on acceptance rate, which is a property of how well the draft imitates the target on your text.

What raises acceptance

Same family, same tokeniser. A 0.5B and an 8B from the same release share training data and vocabulary and typically accept at 0.7 to 0.85. Two models from different families accept poorly and may not share a vocabulary at all, which makes speculation impossible rather than merely slow.

Predictable text. Code, structured output, boilerplate and formatting accept extremely well (often above 0.9), because most tokens are forced by syntax. Free-form creative prose accepts worst. This is why speculative decoding is reported as transformative by people generating code and as marginal by people writing essays, and both reports are correct.

Low temperature. Greedy decoding accepts far more than temperature 1.2, because the target’s own choice is more predictable.

Draft-free variants

A separate draft model costs memory you may not have. Two alternatives:

N-gram or prompt lookup. Search the prompt and recent output for a repetition of the last few tokens, and propose whatever followed last time. Costs nothing, no extra weights, and works remarkably well when the output quotes the input: summarising, editing a document, rewriting code with small changes. Useless on novel text.

Self-speculation. Run a subset of the model’s own layers as the draft. No extra model to store, moderate acceptance.

EAGLE and similar trained heads. A small head trained on the target model’s own hidden states to predict its next tokens. Highest acceptance of the practical methods, needs a trained head for your specific model, and support is engine-dependent.

Turning it on

# llama.cpp: draft model plus target
llama-server -m Qwen3-8B-Q4_K_M.gguf \
  -md Qwen3-0.6B-Q4_K_M.gguf --draft-max 5 --draft-min 1 -ngl 99

# vLLM: n-gram speculation, no draft model
vllm serve <model> --speculative-config \
  '{"method": "ngram", "num_speculative_tokens": 5, "prompt_lookup_max": 4}'

Measure before and after on your own prompts. The gain varies by a factor of three across workloads, and the published figure was measured on somebody else’s.

The honest catch

Speculation helps single-stream latency and does little for aggregate throughput on a busy server. The reason is direct: it exploits idle compute during memory-bound decoding, and batching has already claimed that idle compute for other users. At batch 64 the machine is compute-bound and there is no headroom left to speculate into. Worse, the rejected tokens are wasted compute you did pay for.

So: excellent for one person wanting a fast assistant, or for a latency-sensitive single-user service. Not the answer for a server with many concurrent requests, where the same effort spent on batching and prefix caching pays far better.