Speculative decoding: a small model guesses, a big model checks

The idle machine

Decoding is memory-bound. Each step reads every weight of the model to produce a single token, and the arithmetic units spend most of their time waiting. That means checking five tokens costs barely more than checking one: the weights are read once either way.

Speculative decoding exploits this directly, and it is one of the few optimisations that is free of trade-offs.

The algorithm

1. A small, fast draft model generates the next k tokens, say four. This is cheap: a 1B draft model is roughly a tenth of the cost of a 70B target model per token.
2. The large target model processes all four in a single forward pass, producing its own distribution at each of those positions, which it can do because the causal mask means each position’s output depends only on what precedes it.
3. Compare. Accept the drafted tokens as long as the target model agrees, by a rule that provably preserves the target’s distribution. At the first disagreement, discard the rest and take the target’s own token.
4. Repeat.

If all four are accepted you have produced five tokens for the cost of one target pass plus four cheap draft passes. If none are, you have produced one and wasted a little.

The guarantee that makes it acceptable

The acceptance rule is the clever part. It is not “accept if the target’s top choice matches”. It is a rejection-sampling procedure that accepts or rejects each drafted token with a probability derived from both models’ distributions, and on rejection samples from an adjusted distribution.

The result, proved in the original work, is that the output distribution is identical to sampling from the target model alone. This is not an approximation and it is not a quality trade. Anyone telling you speculative decoding makes output slightly worse has misunderstood it or is describing a different technique.

Why speed-ups are what they are

Typical reported gains are two to three times. The limit is the acceptance rate: how often the small model guesses what the big one would have said.

Acceptance is high on predictable text: boilerplate, code with obvious continuations, the second half of common phrases, structured formats. It is low exactly where the target model is doing something the draft model cannot: unusual reasoning, rare facts, creative phrasing.

So the speed-up varies by content, which is worth knowing when you benchmark: measuring on repetitive text will flatter the technique, and measuring on hard reasoning will understate it.

Variants you will meet

Self-speculation. Use the target model’s own early layers, or a few extra prediction heads bolted on, as the draft. No second model to host. Medusa and EAGLE are the well-known families here.

N-gram or prompt lookup. For summarisation, editing and retrieval-augmented answers, much of the output is copied from the input. Draft by looking up the next few tokens from the prompt where the recent text matches. There is no draft model at all, it is nearly free, and on copy-heavy tasks it works surprisingly well.

Multi-token prediction. Train the model from the start to predict several future tokens at once, giving it a built-in draft. Some recent large models include this.

What it means for you

If you use a hosted API, this is likely already happening and it is part of why per-token latency improved without model changes. You cannot control it and you do not need to.

If you self-host, it is one of the cheaper wins available. vLLM and llama.cpp both support speculative decoding, both are free, and the requirement is a small model sharing the target’s tokenizer: usually the smallest member of the same model family.

The deeper point is the one to keep. A large part of what looks like models getting faster is not better models at all. It is serving engineering (batching, caching, paged memory, speculation) squeezing an inefficient workload. Knowing which is which stops you from attributing an infrastructure improvement to a capability improvement.