Speculative Decoding: Draft Models, EAGLE, and How to Actually Use It
LLM inference is memory-bandwidth bound, not compute bound. The GPU’s tensor cores sit largely idle during autoregressive generation because each forward pass reads the full model weight matrix to produce a single token. A 70B model in BF16 occupies roughly 140 GB; generating one token requires streaming all 140 GB through the GPU’s high-bandwidth memory. With A100 SXM at ~2 TB/s, that caps token throughput at around 14 tokens/second per request before you account for the KV cache, attention, and other overhead. More memory bandwidth does not fundamentally solve the problem—it just shifts the ceiling.
Speculative decoding sidesteps this by reframing generation as a verification problem. A cheap draft model generates several candidate tokens in rapid succession; the expensive target model verifies all of them in a single forward pass. If the draft guesses correctly, you get multiple tokens for the price of one target pass. If it guesses wrong, you fall back gracefully with a mathematically guaranteed correction. The output distribution is identical to sampling from the target model alone—the speedup is strictly free.
Two to four times faster. Lossless. Available today in vLLM and llama.cpp. And yet most production deployments still do not use it. This post is about why that is, and how to fix it.
Why Autoregressive Decoding Is Slow
Transformer inference has two distinct phases. The prefill phase processes the entire prompt in parallel—all tokens computed simultaneously using matrix multiplications that saturate GPU compute. This phase is compute-bound, and GPUs are good at it.
The decode phase generates tokens one at a time. Each step requires a full forward pass through all model layers, attending over the KV cache, and sampling one token. Because each step depends on the previous step’s output, decode cannot be parallelized across tokens. It is fundamentally sequential.
The arithmetic explains why this hurts. Modern GPUs have a high compute-to-memory-bandwidth ratio (arithmetic intensity). Matrix multiplications during prefill achieve thousands of FLOP/byte—well above the hardware’s ridge point—so the GPU runs at peak FLOP utilization. During decode with a batch size of one, each weight matrix is used to multiply a single vector. The arithmetic intensity drops to roughly 2 * d_model FLOP per weight byte (one multiply-accumulate per weight). For Llama 3.1 70B with d_model = 8192, that is about 16,384 FLOP/byte—still well below the A100’s ridge point of roughly 200 FLOP/byte. The GPU is reading weights from memory faster than it can use them.
Prefill (batch of tokens):
Weights --[stream]--> Compute
W * X_batch → highly parallel, compute-bound, ~70% MFU
Decode (one token at a time):
Weights --[stream]--> Compute
W * x_single → memory-bandwidth bound, ~5-15% MFU
Each token = full model weights traversal = ~140 GB read for 70B BF16
Speculative decoding’s insight is that you already have to read all the weights once to verify the draft. If you can verify k draft tokens in that single pass—returning k output tokens instead of one—you have achieved kx throughput at the same memory-bandwidth cost.
The Algorithm
The core procedure is straightforward. Let M_p be the target (large) model and M_q be the draft (small) model.
Step 1 — Draft. Run M_q autoregressively for k steps to produce k candidate tokens x_1, ..., x_k along with their draft probabilities q(x_i | context, x_1, ..., x_{i-1}). Because M_q is much smaller, this is fast.
Step 2 — Verify. Run M_p on the original context plus all k draft tokens in a single batched forward pass. This produces the target model’s logits at each of the k+1 positions (the k draft positions plus the position after the last draft token).
Step 3 — Accept or reject. For each draft token x_i, compute the target model’s probability p(x_i) and the draft model’s probability q(x_i). Accept x_i with probability min(1, p(x_i) / q(x_i)). If x_i is rejected, sample a correction token from a normalized residual distribution and stop.
Step 4 — Bonus token. Even when all k draft tokens are rejected (worst case), the target model’s forward pass has already computed logits at position k+1, so you always emit at least one token per round—never worse than standard decoding.
The key property is that the acceptance/rejection rule is a form of rejection sampling. Accepted tokens come from the target distribution. Rejected tokens are replaced by a correction drawn from max(0, p - q) / Z, which also matches the target distribution. The result: the joint distribution over the entire generated sequence is identical to what M_p would have produced sampling autoregressively. No approximation, no output degradation.
Round with k=4 draft tokens:
Input context: [t_0 t_1 t_2 ... t_n]
Draft pass (M_q, fast):
t_{n+1}=A (q=0.72)
t_{n+2}=B (q=0.65)
t_{n+3}=C (q=0.81)
t_{n+4}=D (q=0.44)
Verify pass (M_p, one forward pass over all 4 tokens):
p(A)=0.68 → accept with prob min(1, 0.68/0.72) = 0.94
p(B)=0.70 → accept with prob min(1, 0.70/0.65) = 1.00
p(C)=0.60 → accept with prob min(1, 0.60/0.81) = 0.74
p(D)=... (not reached—C rejected)
Result: accepted [A, B], rejected C, sample correction token C'
Tokens emitted this round: A B C' (3 tokens, 1 target pass)
The speedup depends entirely on how often the draft model is right. If α is the average per-token acceptance rate, the expected number of tokens emitted per round is (1 - α^{k+1}) / (1 - α). At α = 0.8 with k = 4 draft tokens, you emit on average (1 - 0.8^5) / 0.2 ≈ 3.36 tokens per target pass. The wall-clock speedup is lower because the draft pass has nonzero cost, but empirically this yields 2–3x.
Draft Model Strategies
Different approaches to sourcing draft tokens represent fundamentally different engineering trade-offs.
Independent Small Model
The original speculative decoding paper used a separate smaller model—e.g., a 7B draft for a 70B target. This works well when a capable smaller model exists in the same family (same tokenizer, similar distribution). Llama 3.2 3B drafting for Llama 3.1 70B is a natural pairing.
The downside is that a good draft model must be aligned with the target’s distribution. A Mistral 7B draft for a Llama 70B target will have low acceptance rates due to architectural and training differences. Distribution mismatch is the primary failure mode.
VRAM overhead is real. A 7B BF16 draft model costs 14 GB alongside the 70B target. For a 4-GPU serving setup, this can force you to use a smaller KV cache or quantize.
n-gram Lookahead (Prompt Lookup)
For tasks involving summarization, editing, or RAG—where output tokens are often verbatim copies of input tokens—the prompt itself is the draft model. The algorithm scans the prompt for the last n tokens and looks for matching n-gram continuations, proposing them as draft tokens.
Zero VRAM overhead. Zero draft inference cost. Works extremely well on:
- Summarization (output sentences often mirror input)
- Code completion where function signatures repeat
- Translation with shared proper nouns
Useless for open-ended generation where the output diverges from the input.
EAGLE: Extrapolation Algorithm for Greater Language-model Efficiency
EAGLE (2024) takes a fundamentally different approach. Instead of training a separate small model, it trains a single transformer layer that operates on the target model’s own hidden states. The draft model predicts the target’s next hidden state using the current hidden state plus the current token embedding, then runs the target’s LM head to get draft logits.
EAGLE draft architecture:
[hidden state h_t] ──┐
[token embed e_t] ──┤──> FC + norm + single attn layer ──> ĥ_{t+1}
│
└──> target LM head ──> draft logits
The draft model shares the target's LM head (frozen).
Only the small FC + attention layer is trained.
Because the draft operates in the target’s representation space—predicting hidden states rather than logit distributions—alignment is high. EAGLE achieves acceptance rates in the 0.78–0.85 range on code and chat tasks, translating to 2.5–3.5x speedup.
EAGLE draft models are small. For a 70B target, the EAGLE draft head adds roughly 1–2B parameters. The VRAM overhead is minor compared to a full small-model draft.
EAGLE2 (mid-2024) added dynamic draft trees. Instead of always proposing k tokens in a linear sequence, EAGLE2 builds a branching tree of candidates, scoring each branch by confidence and pruning low-probability paths. The target verifies the entire tree in one pass by treating the tree as a batch of parallel sequences. This improves acceptance rates by 20–40% over EAGLE1 at the same compute budget, reaching speedups of 3.05–4.26x on some benchmarks.
EAGLE2 draft tree (k=4, branching):
Position n+1: [A(0.72)] [B(0.18)]
Position n+2: [C(0.65)] [D(0.22)] [E(0.78)]
Position n+3: [F(0.71)] [G(0.54)] [H(0.80)]
High-confidence paths (product of branch probs) retained.
Low-probability branches pruned before verification.
Target model verifies all remaining nodes in one batched pass.
EAGLE3 (2025) introduced multi-feature drafting—using hidden states from multiple layers of the target, not just the final layer, to give the draft model richer signal. This reduces the acceptance-rate gap between draft and target further, particularly on complex reasoning tasks where single-layer hidden states are less predictive.
Medusa
Medusa adds multiple independent prediction heads to the target model itself—each head predicts a token k positions ahead. The heads are cheap (one linear layer per head) and trained on top of the frozen target. Verification works by checking all heads’ predictions simultaneously.
Medusa trades acceptance rate for simplicity: it does not use the draft model’s previous outputs when predicting further positions, so later positions are harder to predict accurately. Acceptance rates are generally lower than EAGLE2, but there is no separate draft model to manage.
Self-Speculative Decoding
Rather than adding a separate model, self-speculative decoding skips certain layers in the target model itself for the draft pass, then runs the full model for verification. A 70B model drafting with layers 0–32 and verifying with all 80 layers. This works because early-exit representations are often predictive of the final output distribution.
The appeal is operational simplicity: one model, one set of weights. The speedup is typically lower (1.5–2x) because the draft is still relatively expensive and acceptance rates are moderate.
When Speculative Decoding Helps (and When It Doesn’t)
Speculative decoding’s speedup is not free—it degrades to slightly worse than standard decoding when the draft acceptance rate is very low.
Where It Excels
Low batch sizes. At batch size 1 (real-time chat, coding assistants), the GPU is severely underutilized during decoding. Speculative decoding increases effective batch size by verifying multiple draft tokens per pass, recovering utilization. Speedups of 2–4x are realistic.
Code generation. Code is highly predictable. Given def calculate_total(items):, the next several tokens are nearly certain. EAGLE and ngram both achieve high acceptance rates on code.
Tasks with input-output overlap. Summarization, RAG answers, document editing. Ngram lookup is particularly effective here.
Long outputs. The speedup compounds per token. A 2x speedup on a 2000-token generation saves 1000 target passes.
Where It Underperforms
Large batch sizes. At batch size 32+, the GPU is already well-utilized during decode. Adding speculative decoding increases per-batch latency slightly (more tokens to verify). Throughput improvement diminishes or reverses. Most published benchmarks measure latency at batch size 1—a real serving deployment at high throughput may see minimal benefit.
High-temperature sampling. At temperature 1.5+, the target model’s distribution is flat. The draft model’s tokens are less likely to match, acceptance rates fall, and the overhead of running the draft dominates.
Tasks requiring creative divergence. Open-ended creative writing where the next token is genuinely unpredictable. The draft will be wrong frequently enough that the per-round overhead dominates.
Models with no published draft. EAGLE2 requires a trained draft head. If no one has released an EAGLE2 draft for your specific model, you need to train one (not trivial) or fall back to ngram/small-model approaches.
Speculative decoding benefit by scenario:
Scenario | Draft type | Expected speedup
----------------------------|-----------------|------------------
Batch-1 chat, temp=0.7 | EAGLE2 | 2.5–3.5x
Batch-1 code completion | EAGLE2 or ngram | 3–4x
RAG answers | ngram | 2–3x
Batch-32 chat | EAGLE2 | 1.1–1.4x
Batch-128 serving | any | <1.1x or negative
High-temperature creative | any | 0.9–1.2x
Model with no EAGLE draft | ngram | 1.5–2.5x (prompt-overlap tasks)
vLLM Configuration
vLLM’s speculative decoding support is mature as of v0.7+ and covers all major draft strategies. Configuration is handled via --speculative-config at the CLI or speculative_config in the AsyncLLMEngine constructor.
EAGLE / EAGLE2
EAGLE draft models for common architectures are published on HuggingFace by the EAGLE team (yuhuili/) and by community contributors. Check for your specific model before assuming one exists.
|
|
|
|
The num_speculative_tokens parameter (k in the algorithm) trades draft overhead against potential speedup. More tokens means more potential gain per round but more wasted work when acceptance fails. In practice, 4–6 is the sweet spot for EAGLE on 7B–70B models.
n-gram Lookahead (Prompt Lookup Decoding)
No external model needed. Ideal when output is likely to mirror input—summarization, editing, translation, RAG.
|
|
prompt_lookup_min is the minimum n-gram size to match (2 = bigrams). prompt_lookup_max is the maximum. Higher values find more specific matches but find them less frequently. Start with min=2, max=5 and tune based on your task.
Draft Model (Independent Small Model)
|
|
Both models must use the same tokenizer. Ensure the draft model was trained on a similar distribution—instruct draft for instruct target, base for base.
Medusa
Medusa heads are separate trained artifacts. If none exist for your model, skip this.
|
|
Monitoring Acceptance Rate
vLLM exposes Prometheus metrics for speculative decoding. The acceptance rate is the single most important metric—if it falls below ~0.6, the draft is costing more than it gains.
|
|
|
|
llama.cpp Configuration
llama.cpp has supported basic speculative decoding (independent draft model) for some time. EAGLE3 support landed in late 2025 but lagged behind vLLM and TRT-LLM; check the discussion tracker if you need cutting-edge draft methods.
Draft Model
|
|
The --draft N flag sets how many tokens the draft model proposes per round (equivalent to k). The -md flag specifies the draft model path.
For CPU inference, speculative decoding is less beneficial because the bottleneck is different—but it can still help when the draft model fits in faster memory (L3 cache, RAM) while the target is memory-bandwidth-limited.
n-gram (Prompt Lookup) in llama.cpp
|
|
Measuring Performance in llama.cpp
|
|
Training an EAGLE Draft Head
If no pre-trained EAGLE head exists for your model, you can train one. The training cost is modest—a single GPU for 1–2 days on a standard dataset.
|
|
The 50k sample dataset is the minimum viable training run. 200k samples meaningfully improves acceptance rate, especially on domain-specific content that differs from the training distribution.
If you are serving a fine-tuned model (instruction tuning, domain adaptation), consider training a dedicated EAGLE head on the fine-tuned model rather than the base. The acceptance rate gap between a base-model EAGLE head and a fine-tuned target can be 15–25 percentage points.
Production Considerations
Batch Size is the Critical Variable
Published speedup numbers are almost always measured at batch size 1. Real production systems handle concurrent requests. As batch size grows, the GPU becomes better utilized during standard decode, and the relative benefit of speculative decoding shrinks.
The crossover point depends on model size and hardware. On a 4xA100 serving a 70B model, speculative decoding typically remains beneficial up to batch size 8–12 for latency metrics. For throughput metrics (tokens/second at steady state), the benefit disappears earlier—often by batch size 4–8.
If your deployment is latency-constrained (interactive chat, coding assistant), speculative decoding is worth enabling even at moderate batch sizes. If it is throughput-constrained (bulk processing), benchmark carefully before committing.
Speculative decoding latency impact by batch size (approximate, 70B EAGLE2):
Batch size | Token latency (no spec) | Token latency (spec) | Speedup
-----------|-------------------------|----------------------|--------
1 | 52 ms/tok | 17 ms/tok | 3.1x
2 | 55 ms/tok | 20 ms/tok | 2.7x
4 | 60 ms/tok | 26 ms/tok | 2.3x
8 | 68 ms/tok | 38 ms/tok | 1.8x
16 | 80 ms/tok | 58 ms/tok | 1.4x
32 | 95 ms/tok | 80 ms/tok | 1.2x
64 | 130 ms/tok | 118 ms/tok | 1.1x
VRAM Budget
EAGLE draft heads are small (1–2B effective parameters for a 70B target draft head). A dedicated 8B EAGLE draft model costs roughly 16 GB in BF16. This may require reducing --max-model-len or adding a GPU.
For EAGLE draft heads specifically (not full small models), the overhead is much lower—typically 2–4 GB for the draft head alone when the target model’s LM head is shared.
VRAM overhead by draft approach (70B BF16 target):
Method | Draft VRAM | Notes
--------------------|-------------|------------------------------------
ngram | 0 GB | No model; uses prompt buffer
EAGLE head | ~2–4 GB | Shared LM head with target
7B draft model | ~14 GB | Full separate model
13B draft model | ~26 GB | Full separate model
Self-speculative | 0 GB extra | Same model, skip layers
Temperature and Sampling Parameters
Speculative decoding is compatible with any sampling strategy—temperature, top-p, top-k—because the acceptance/rejection step adjusts for the draft model’s distribution regardless of sampling configuration. You do not need to change your sampling parameters.
At very high temperatures (>1.3), acceptance rates fall and the benefit diminishes. This is a property of the task, not a bug.
Greedy Decoding (Temperature=0)
Speculative decoding with greedy decoding (temperature=0) is simpler: a draft token is accepted if and only if it is the argmax of the target model’s distribution at that position. Acceptance rates tend to be higher under greedy decoding because the target distribution is deterministic.
|
|
Comparing Approaches
Draft strategy comparison:
Method | Speedup | Acceptance | VRAM | Draft needed | Notes
----------------|----------|------------|---------|--------------|------------------------
ngram | 1.5–3x | varies | none | no | Best for summarization/RAG
Small model | 1.5–2.5x | 0.65–0.75 | high | yes | Easy if same family exists
EAGLE1 | 2–3x | 0.78–0.83 | low | yes (train) | Linear draft tree
EAGLE2 | 2.5–4x | 0.80–0.88 | low | yes (train) | Dynamic tree, best overall
EAGLE3 | 2.5–4.5x | 0.82–0.90 | low | yes (train) | Multi-feature, best accuracy
Medusa | 1.5–2.5x | 0.60–0.75 | medium | yes (train) | Simple; lower acceptance
Self-speculative| 1.3–2x | varies | none | no | Elegant; modest gains
Getting Started Quickly
The fastest path to measurable speedup with zero new training:
1. Check if an EAGLE draft exists for your model. Search HuggingFace for EAGLE {your-model-name}. If found, use it with vLLM as shown above.
2. If not, use ngram lookup for your use case. Summarization, RAG, document editing—enable ngram with prompt_lookup_min=2, prompt_lookup_max=5.
3. Benchmark before deploying. Run with and without speculative decoding at your production batch size. If the acceptance rate metric from Prometheus is below 0.6, reconsider the draft strategy.
4. Consider training an EAGLE head if your workload justifies it. If you are serving millions of tokens per day on a custom fine-tuned model with no published draft, the training cost (1–2 GPU-days, 50k samples) pays back quickly.
The common reason teams do not use speculative decoding in production is the assumption that it requires significant setup. For EAGLE with a published draft model and vLLM, the entire configuration is a JSON object added to an existing vllm serve invocation. There is no operational complexity beyond confirming the acceptance rate metric looks healthy.
The output is identical to standard decoding. The cost per token drops by 2–4x. Enable it.
Further Reading
- EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty — original EAGLE paper
- EAGLE-2: Faster Inference with Dynamic Draft Trees
- vLLM Speculative Decoding Documentation
- EAGLE Draft Models on HuggingFace — pre-trained heads for popular models
- llama.cpp speculative decoding examples
- Speculative Decoding: Exploiting Speculative Execution for Accelerating Seq2seq Generation — original 2022 paper
Comments