Speculative Decoding Explained: Trading Compute for Bandwidth to Decode Faster

Autoregressive decoding has a structural problem that no amount of better hardware fixes on its own: it produces exactly one token per forward pass, and each forward pass drags every weight in the model out of high-bandwidth memory whether you are computing one token or a thousand. A 70B model in FP16 is 140GB of weights; emitting a single token means reading all 140GB from HBM, doing a comparatively trivial amount of arithmetic, and throwing the loaded weights away. The math units sit mostly idle. The bottleneck is not FLOPs — it is memory bandwidth, and a single decode step uses a rounding error’s worth of the chip’s compute. Speculative decoding is the trick that spends that idle compute to buy back latency: let a cheap model guess several tokens ahead, then have the expensive model check all the guesses in one pass that costs almost the same as checking one. When the guesses are good, you get three or four tokens for the price of a single forward pass, and — this is the part that makes it usable in production rather than a research toy — the text that comes out is provably identical in distribution to what the big model would have produced on its own.

This post is about how that works, why it is lossless rather than an approximation, the single number that determines whether you get a 1.2x or a 3x speedup, where the draft predictions actually come from in 2026 systems, and the regime — high-batch serving — where speculation can quietly make your throughput worse.

The Core Trade: Idle Compute for Memory Bandwidth

Everything about speculative decoding follows from one fact about the decode phase of inference, the same fact that dominates KV cache engineering: generating tokens one at a time is memory-bandwidth-bound, not compute-bound. During prefill, the model processes the whole prompt in parallel and the GPU’s tensor cores run hot. During decode, each step processes a single new token, so the matrix multiplies degenerate into matrix-vector products. The arithmetic intensity collapses, and the GPU spends its time waiting on weight reads from HBM rather than on the multiply-accumulate units.

The crucial consequence: a forward pass that scores K tokens at once costs almost the same wall-clock time as a forward pass that scores one token. The weights are read from memory exactly once either way; the only added work is K times more arithmetic against those already-loaded weights, and arithmetic is the resource you have in surplus. Scoring five candidate tokens in parallel is nearly free relative to scoring one.

That asymmetry is the entire opportunity. If some cheap process can propose the next five tokens, the expensive model can verify all five in a single bandwidth-bound pass. Every verified token that turns out to be correct is a token you generated without paying for its own forward pass. You have converted the model’s wasted compute headroom into reduced latency.

The cheap proposer is the draft model; the expensive model being accelerated is the target model. The whole game is making the draft good enough and cheap enough that the target’s near-free parallel verification pays for itself.

The Algorithm: Draft, Verify, Accept

A single step of speculative decoding looks like this. The draft model runs autoregressively to propose a short sequence of γ tokens (commonly 3 to 8). The target model then runs once, in parallel, over the current token plus all γ drafted tokens, producing its own probability distribution at each of those positions. You walk left to right comparing the draft’s tokens against the target’s distributions, accepting the longest prefix the target agrees with, and the moment there is a disagreement you discard the rest of the draft and emit one corrected token from the target itself.

One step = ONE expensive target forward pass:

draft model (cheap, runs autoregressively):
   ctx ─► d1 ─► d2 ─► d3 ─► d4 ─► d5        propose γ=5 tokens

target model (expensive, ONE parallel pass over all positions):
   [ ctx  d1   d2   d3   d4   d5 ]
            │    │    │    │    │
            ▼    ▼    ▼    ▼    ▼
   verify  d1   d2   d3   d4   d5   + free sample at first reject

accept:    d1   d2   d3   ✗               d4 disagrees → stop
emit:      d1 + d2 + d3 + one corrected token  =  4 tokens this step

The payoff is visible in the last line: this step emitted four tokens but cost one target forward pass (plus the cheap draft passes). On a step where the draft is fully wrong, you accept zero drafted tokens and still emit the single token the target would have produced anyway — so the worst case degrades to ordinary autoregressive decoding plus the wasted draft work, never to fewer tokens than baseline. On a step where the draft is fully right, you emit γ + 1 tokens for one target pass.

Notice the “free sample at first reject.” Because the target already computed its distribution at the rejection position, you sample the corrected token from that distribution at no extra cost. That is why a step always advances by at least one token even when nothing is accepted: you never come away from a target pass empty-handed.

Why It Is Lossless: The Rejection-Sampling Step

The objection every engineer raises first is the obvious one: if a small dumb model is choosing tokens, isn’t the output quality just the small model’s quality? No — and the reason is the single most elegant part of the technique. Speculative decoding does not accept the draft’s tokens because they are good; it accepts them with a precisely calibrated probability that makes the combined process sample from the target model’s distribution exactly. The draft model is a proposal mechanism, not a decision-maker.

Concretely, for a drafted token x at a position where the draft assigned probability q(x) and the target assigns p(x), you accept x with probability:

accept_prob = min(1, p(x) / q(x))

If the target likes the token at least as much as the draft did (p(x) ≥ q(x)), you always accept. If the target likes it less, you accept proportionally and, on rejection, resample the replacement token from the residual distribution norm(max(0, p − q)) — the part of the target’s distribution that the draft under-weighted. This is rejection sampling, and the theorem (proved independently by Leviathan et al. and Chen et al. in 2023) is that the resulting token is distributed exactly according to p. Over a whole sequence, the speculative process and plain sampling from the target are statistically indistinguishable.

The practical implications are worth stating plainly:

• Greedy decoding is bit-exact. At temperature 0, “the target agrees” means the drafted token is the target’s argmax, and the output is identical to non-speculative greedy decoding token for token.
• Sampling is distribution-exact. At temperature > 0 you will not get the same tokens as a non-speculative run with the same seed, but you get a valid sample from the same distribution. Quality is not traded away.
• It is not an approximation you tune for accuracy. Unlike quantization, there is no quality knob. The acceptance math is what it is; what you tune (γ, the draft model) only affects speed.

One caveat: some production systems offer a faster “lossy” or “typical acceptance” mode that relaxes the exact criterion to raise the acceptance rate. That genuinely changes the output distribution. The default in serious serving stacks is the exact version above, and you should know which one you have enabled before claiming your outputs are unchanged.

The Acceptance Rate Is the Whole Ballgame

The speedup of speculative decoding is governed almost entirely by one quantity: the acceptance rate α, the probability that a drafted token survives verification. It is a property of how well the draft model mimics the target on your actual workload, and it ranges from near zero (mismatched draft, out-of-distribution input, high temperature) to above 0.9 (a well-trained EAGLE head on in-distribution text).

With acceptance rate α and γ drafted tokens per step, the expected number of tokens emitted per step is the closed form from the Leviathan paper:

E[tokens per step] = (1 − α^(γ+1)) / (1 − α)

That is the gross speedup before accounting for draft cost. The table below shows it for γ = 5:

But gross tokens-per-step is not the speedup, because the draft is not free. The draft runs γ autoregressive passes of its own. If the draft model costs a fraction c of a target forward pass per token, the real cost of a step is roughly 1 + γ·c target-pass-equivalents (one verification pass plus the draft work), and the net speedup is:

speedup ≈ E[tokens per step] / (1 + γ·c)

Plug in realistic numbers. A draft model 1/20th the size of the target gives c ≈ 0.05, so with γ = 5 the cost per step is about 1 + 0.25 = 1.25. At α = 0.8 that is 3.69 / 1.25 ≈ 2.95x; at α = 0.7, 2.94 / 1.25 ≈ 2.35x. That is exactly the “2-3x” range people quote — and it is now obvious what drives it: a draft that is both cheap (small c) and accurate (high α). Those two pull in opposite directions, because a bigger draft model is more accurate but less cheap, which is precisely the tension that the learned-drafter methods below exist to resolve.

Where the Draft Comes From

The earliest speculative systems used an independent draft model — a small model from the same family as the target, sharing its tokenizer: Llama-3.2-1B drafting for Llama-3.1-70B, for instance. This works, but it has two problems. The draft consumes its own VRAM and its own forward passes, and a generically-trained small model only loosely tracks the target’s distribution, capping α. The field has since moved toward drafters that are trained or structured specifically to predict the target’s own next tokens, which raises acceptance dramatically while shrinking draft cost.

Prompt lookup decoding (also called n-gram speculation) deserves special mention because it is almost embarrassingly cheap and shockingly effective on the right workload. There is no draft model at all; the “draft” is simply the longest matching n-gram copied from the existing context. For tasks where the output echoes the input — summarization, code completion where you are editing existing code, retrieval-augmented generation that quotes sources, structured-output reformatting — large spans of the response already appear in the prompt, and copying them verbatim achieves very high acceptance for zero model cost. It does nothing for open-ended creative generation, where the output shares little with the input.

Medusa bolts several extra prediction heads onto the target model itself. Head 1 predicts the token after next, head 2 the one after that, and so on. Because the heads share the target’s backbone, there is no separate model to host, and Medusa verifies a tree of candidate continuations (more on that below) in a single pass. Medusa-1 freezes the backbone and trains only the heads; Medusa-2 fine-tunes jointly for higher acceptance.

EAGLE is the method most current high-quality serving stacks default to. Its insight is that predicting tokens directly is hard, but predicting the target’s second-to-top-layer feature vector is much easier and more regular, because that feature space is smoother than the token space. EAGLE runs a small autoregressive head at the feature level, feeding back both the sampled token’s embedding and the predicted feature, then maps to a token. EAGLE-2 made the draft tree dynamic (expanding branches where the head is confident), and EAGLE-3 scaled the approach further. The result is acceptance lengths of three-to-five-plus tokens at a draft cost far below an independent model — the best α-versus-c trade available without training a whole second model.

Tree Speculation: Verifying Many Guesses at Once

A single linear draft sequence throws away the parallel-verification budget. The target pass can score far more than γ positions for nearly the same cost, so why commit to one guess per position? Tree-based speculation drafts multiple candidate branches — several plausible tokens at each step, fanning out into a tree of possible continuations — and verifies the entire tree in one target forward pass using a specially constructed tree attention mask that lets each candidate path attend only to its own ancestors.

Linear draft (1 path):        Tree draft (many paths, 1 verify pass):

ctx─d1─d2─d3─d4─d5             ctx ─┬─ "the" ─┬─ "cat"
                                    │         └─ "dog"
                                    └─ "a"  ──── "bird"
   verify 5 positions            verify all nodes; accept the
                                 longest path the target ratifies

Because verification is bandwidth-bound, scoring 30 tree nodes costs nearly the same as scoring 5 linear tokens, and the tree dramatically raises the odds that some path matches the target deep into the future. This is why Medusa and EAGLE-2 both build trees: the expensive resource (a target pass) is fixed, so you stuff as many candidate futures into it as the compute headroom allows and let verification pick the winner. The cost is kernel complexity — tree attention masks and the bookkeeping to reconstruct the accepted path are the gnarly part of any real implementation.

When Speculation Hurts

Speculative decoding is not a free lunch, and the failure modes are specific and predictable.

Large batch sizes are the big one. The entire premise is that the target model has idle compute during decode. That is true at batch size 1 — a latency-sensitive single stream — but as you batch more concurrent requests together, the decode step stops being bandwidth-bound and becomes compute-bound, because now you are doing real matrix-matrix multiplies across the batch and the tensor cores fill up. At that point the “free” parallel verification is no longer free: every speculative token you score is compute that competes with other requests in the batch. Past a crossover batch size, speculation reduces aggregate throughput even as it might still help any single request’s latency. The practical rule is that speculation is a latency optimization for low-concurrency or interactive serving, not a throughput optimization for a saturated batch server. Some schedulers now disable speculation dynamically once batch occupancy crosses a threshold.

Low acceptance wastes everything. If the draft is poorly matched to the workload — wrong domain, an aggressive sampling temperature that makes the target’s distribution flat and unpredictable, genuinely novel content with no n-gram overlap — α collapses, you accept almost nothing, and you have paid full draft cost plus verification overhead to generate tokens at baseline speed or slower. High temperature is a particular acceptance killer: a flatter target distribution means the draft’s confident guess is more often “wrong” by the rejection criterion.

The draft must stay cheap. As the 1 + γ·c term shows, if the draft model creeps up in size (c grows) the denominator swells and the speedup evaporates even at high acceptance. This is the trap of “just use a bigger, more accurate draft model”: you raise α but raise c faster. It is also why EAGLE-style feature-level heads win — they push α up while keeping c tiny.

Memory and complexity. An independent draft model is more weights in VRAM that could otherwise hold KV cache or larger batches. Tree attention kernels and the accept/resample logic are real engineering surface area. None of this is disqualifying, but it is why speculation is a serving-stack feature you enable deliberately, not a default you forget about.

Running It in Production

Every major serving stack supports speculative decoding behind a config block. In vLLM, you attach a speculative_config to the engine. The independent-draft-model form:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

from vllm import LLM, SamplingParams

llm = LLM(
    model="meta-llama/Llama-3.1-70B-Instruct",
    tensor_parallel_size=4,
    speculative_config={
        "model": "meta-llama/Llama-3.2-1B-Instruct",
        "num_speculative_tokens": 5,
    },
)
print(llm.generate("Explain speculative decoding in one sentence.",
                   SamplingParams(temperature=0.0))[0].outputs[0].text)

The same flag from the server CLI, plus the two model-free and learned-drafter variants that need no separate checkpoint to host:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

# Independent draft model
vllm serve meta-llama/Llama-3.1-70B-Instruct \
  --tensor-parallel-size 4 \
  --speculative-config '{"model": "meta-llama/Llama-3.2-1B-Instruct", "num_speculative_tokens": 5}'

# Prompt-lookup / n-gram: zero extra weights, great for summarization & code edits
vllm serve meta-llama/Llama-3.1-70B-Instruct \
  --speculative-config '{"method": "ngram", "num_speculative_tokens": 5, "prompt_lookup_max": 4, "prompt_lookup_min": 2}'

# EAGLE head: highest acceptance, tiny draft cost
vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --speculative-config '{"method": "eagle", "model": "yuhuili/EAGLE-LLaMA3.1-Instruct-8B", "num_speculative_tokens": 5}'

TensorRT-LLM exposes the same family — draft-target, Medusa, EAGLE, and lookahead — through its build and serve configuration, and SGLang ships EAGLE support tuned for its RadixAttention cache. The knobs that matter are the same everywhere: which drafting method, the speculative token count γ (num_speculative_tokens), and, increasingly, whether the scheduler is allowed to turn speculation off under load.

The one measurement that matters once it is running is the acceptance rate / mean accepted length, which vLLM and friends export in their metrics. Watch it on your real traffic, not a benchmark. If your mean accepted length is 1.3, speculation is barely helping and the draft is mismatched to your workload; if it is 4+, you are extracting most of the available win and could try raising γ. And always validate the regime: measure end-to-end tokens/second at your actual serving batch size, because a configuration that is 3x faster at batch 1 can be net-negative at batch 64.

Verdict

Speculative decoding is one of the rare optimizations that is both principled and free of quality cost: it exploits the structural idle compute of bandwidth-bound decoding, and the rejection-sampling step guarantees the output is a true sample from the target model — bit-exact under greedy decoding. For interactive, latency-sensitive serving at low concurrency, it is close to mandatory in 2026; a 2-3x latency reduction with no accuracy trade is not a knob you leave off. The method to reach for is EAGLE (or a tree-based learned drafter) for general text and prompt-lookup/n-gram for input-echoing workloads like summarization, RAG, and code editing, where it costs nothing and acceptance is high.

The honest caveats are about regime, not quality. The speedup lives and dies by acceptance rate times draft cheapness, both measured on your workload, and the whole premise inverts at high batch sizes where the target model is already compute-bound — there speculation can cost you aggregate throughput. Treat it as a latency tool, instrument the accepted length, let the scheduler disable it under heavy batching, and verify your numbers at production concurrency rather than trusting the headline multiplier. Do that, and it is one of the highest-leverage changes you can make to an inference stack without touching the model itself.

Sources

• Fast Inference from Transformers via Speculative Decoding (Leviathan, Kalman, Matias, 2023)
• Accelerating Large Language Model Decoding with Speculative Sampling (Chen et al., DeepMind, 2023)
• Medusa: Simple LLM Inference Acceleration with Multiple Decoding Heads
• EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty
• EAGLE-2: Faster Inference with Dynamic Draft Trees
• vLLM documentation — Speculative Decoding
• NVIDIA TensorRT-LLM — Speculative Decoding
• Prompt Lookup Decoding (Apoorv Saxena)