Speculative Decoding: Speedup, Variants, and How to Enable It

TL;DR

• What it is. A draft proposes K tokens; the target verifies all K in one forward pass; modified rejection sampling keeps the target's exact output distribution.
• Why it works. Decoding is memory-bandwidth-bound. Verifying K tokens reads the weights once, the same as generating one token.
• How fast. Speedup = (1 - α^γ+1) / ((1 - α)(γc + 1)). At acceptance rate α = 0.8 with a free 5-token draft, 3.69x. Published: 2x-3x (T5-XXL), 2-2.5x (Chinchilla 70B), 2.7x-3.5x (EAGLE, LLaMA2-Chat 70B), 3.05x-4.26x (EAGLE-2), up to 6.5x (EAGLE-3).
• The variants. Separate draft model, n-gram prompt lookup, Medusa heads, EAGLE feature-level drafts, and native multi-token-prediction heads. They differ in where the draft comes from and what it costs to run.
• How to turn it on. vLLM: a JSON --speculative-config. SGLang: --speculative-algorithm plus draft-tree flags. Both below.
• In production. morph-v3-fast runs n-gram speculation with a 64-token draft window at 10,500 tok/s. Morph's open coding models run speculators trained on coding traffic.

New to the idea? How does speculative decoding work? is the plain-language version of this page.

Why Decoding Is Memory-Bound

Autoregressive generation produces one token per forward pass. Each pass loads every layer's weights from GPU memory into the compute cores, then does a small amount of arithmetic on a single token. At batch size one, the weight load dominates. The GPU's math units sit mostly idle while bytes stream in. Decoding is memory-bandwidth-bound, not compute-bound.

The spare compute is the opportunity. Running K candidate tokens through the model in one pass reads the weights once and does K tokens of arithmetic. On a memory-bound GPU that costs almost the same as one token. Leviathan et al. state it directly: parallel scoring of short continuations from a fast draft has latency comparable to sampling a single token from the target.

The Draft-Then-Verify Loop

One round does four things. The draft proposes γ candidate tokens autoregressively, which is cheap because the draft is small or model-free. The target runs one forward pass over the context plus all γ candidates and records its own distribution at each position. Verification compares candidates left to right and accepts each one with probability min(1, p(x)/q(x)), where p is the target and q is the draft. The first rejection stops the run; the target samples a corrected token from the residual distribution, and the next round starts there.

Concretely: the draft proposes 5 tokens, the target accepts the first 3 and rejects the 4th. Those 3 tokens plus the corrected 4th cost one target forward pass instead of four. Across many rounds, the average number of tokens produced per pass is the speedup, minus the draft's own cost.

// gamma = draft length, e.g. 5
const draft = draftModel.propose(context, gamma)          // cheap

// ONE target forward pass over context + all gamma candidates
const targetProbs = targetModel.forward([...context, ...draft])

// Accept left to right with prob min(1, p(x)/q(x)); stop at first reject.
// On reject, sample from norm(max(0, p - q)) so the output distribution
// is exactly the target's (Leviathan et al. 2022, Appendix A.1).
const { accepted, corrected } = speculativeSample(draft, targetProbs)

context.push(...accepted, corrected)   // accepted tokens were ~free

Why It Is Lossless

The accept/reject rule is not a heuristic. Leviathan et al. prove that sampling x from the draft q, keeping it when q(x) ≤ p(x), rejecting it with probability 1 - p(x)/q(x) otherwise, and resampling rejections from norm(max(0, p(x) - q(x))) yields samples distributed exactly as p(x). Chen et al. give the same proof independently. The target model's output distribution is preserved within hardware numerics. Greedy decoding with a greedy draft reduces to exact token matching.

This is why speculative decoding is safe to ship without an eval cycle. Quantization, distillation, and pruning each trade some quality for speed. Speculative decoding trades none. It requires no retraining and no architecture change to the target. Leviathan et al. reported identical outputs on T5-XXL at 2x-3x. Chen et al. reported 2-2.5x on Chinchilla 70B on XSum and HumanEval without compromising sample quality.

Speedup by Acceptance Rate

Two numbers set the speedup. The acceptance rate α is the probability the target accepts a draft token. The draft length γ is how many tokens are proposed per round. A third, the cost coefficient c, is the wall time of one draft step divided by one target step. Leviathan et al.'s Theorem 3.8 gives the expected wall-time improvement:

speedup = (1 - α^γ+1) / ((1 - α)(γc + 1))

Two consequences follow. The ceiling for any draft is 1/(1 - α): at α = 0.9 nothing gets past 10x no matter how long the draft. And γ has an optimum: past it, the extra draft tokens are mostly rejected while their cost γc keeps growing. The table evaluates the formula at c = 0.05, the highest value Leviathan et al. observed with a draft two orders of magnitude smaller than the target, and at c = 0 for a model-free draft such as n-gram lookup.

The paper's own Table 1 checks out against the formula: α = 0.8 with γ = 5 gives 3.69x, and α = 0.9 with γ = 10 gives 6.86x, both at c = 0. Its measured T5-XXL results land inside the table's middle rows: 2.6x at temperature 1 and 3.4x at temperature 0 on English-German translation, 2.3x and 3.1x on summarization. Temperature matters because a hotter target is less predictable, which lowers α.

What the Draft Is Changes the Speedup

The formula makes α the lever, and α is a property of the draft-target pair on a given workload. A generic small model from the same family agrees with the target often enough for 2x-3x. A draft trained on the target's own traffic agrees more often and pushes past 3x. A draft that can read the answer out of the prompt, as in code editing, agrees most often of all.

Leviathan et al. tested the extreme case: a trivial bigram model as the draft for T5-XXL gave α ≈ 0.2 on translation, enough for 1.25x at γ = 3 because the draft cost nothing. That observation is what became n-gram and prompt-lookup speculation. When the output copies long spans of the input, a table lookup drafts them perfectly, and the same free draft that gave 1.25x on translation gives far more on code editing. That is the regime Morph's Fast Apply model lives in.

The Variants: Draft Model, N-gram, Medusa, EAGLE, MTP

Every variant shares the verification step. They differ in where the draft comes from, whether it needs training, and what it costs per round.

Medusa (Cai et al., arXiv 2401.10774) keeps a single model. Extra decoding heads predict several future positions at once, and tree attention verifies many candidate continuations in one pass. Medusa-1 trains only the heads on a frozen target and exceeds 2.2x. Medusa-2 fine-tunes heads and target together for 2.3-3.6x.

EAGLE (Li et al., arXiv 2401.15077) moves drafting to the feature level. A small head autoregresses on the target's second-to-top-layer features, conditioned on the token sequence shifted by one step to resolve feature-level uncertainty. On LLaMA2-Chat 70B it reaches 2.7x-3.5x latency speedup and doubles throughput while preserving the output distribution. EAGLE-2 (arXiv 2406.16858) replaces the static draft tree with a context-aware dynamic one, using the draft head's calibrated confidence to decide where to branch, for 3.05x-4.26x, 20%-40% over EAGLE-1. EAGLE-3 (arXiv 2503.01840) drops feature prediction for direct token prediction with multi-layer feature fusion, trained with a technique the authors call training-time test, for up to 6.5x, about 1.4x over EAGLE-2, and a 1.38x throughput gain at batch size 64 inside SGLang.

DSpark (DeepSeek and Peking University, 2026) is the most recent shape of the idea and the one vLLM's adaptive-verification path currently targets. A parallel backbone drafts a run of tokens with confidence scores, a hardware-aware scheduler keeps the prefix likely to survive, and the target verifies only that prefix. The loop is unchanged; the scheduler decides how much of it to run.

Published Speedups

Every number below is from the cited paper, README, or serving-framework documentation. Speedup ratios are relative to vanilla autoregressive decoding of the same target model on the same hardware.

The SGLang row is the cleanest apples-to-apples measurement: same model, same GPU, same benchmark, three configurations. EAGLE-3 gives 2.36x over the baseline on that setup, which sits at the α ≈ 0.7-0.8 rows of the acceptance-rate table above.

Speculative Decoding in Production at Morph

Morph serves inference for AI coding agents, and code is the workload where speculative decoding pays off most. Two production uses illustrate the two ends of the variant table.

Fast Apply, n-gram speculation. morph-v3-fast merges an edit snippet into a file and returns the full merged file. Most output tokens already exist in the input, so a prompt-lookup draft that copies the next 64 tokens from the original file is right almost every time it fires. That is the c = 0 column of the acceptance-rate table with a high α. Combined with continuous batching and custom kernels, morph-v3-fast serves at 10,500 tok/s. The same family of tricks compresses context with Compact at 33,000 tok/s.

Open coding models, trained speculators. General chat traffic has no input to copy from, so the open models Morph serves (Kimi K3, GLM-5.3, GLM-5.3-Flash, DeepSeek V4 Flash) run with draft heads trained on coding-agent traffic. A speculator trained on the same distribution it will see in production has a higher α than a generic one, which is the "draft tuned to your workload" bar in the comparison above. On private deployments with a custom speculator, DeepSeek V4 Flash reaches up to 150 tok/s per user.

Dedicated deployments train the speculator on your traffic. See dedicated inference and the benchmark evidence behind the per-user speed numbers.

How to Enable It: vLLM and SGLang

Both major open-source serving stacks ship speculative decoding behind launch flags. The flags below are taken from the current vLLM and SGLang documentation as of 2026-09-01.

vLLM

vLLM takes a single JSON object via --speculative-config (or speculative_config= on LLM(...)). The method key selects the variant: draft_model, ngram, suffix, mtp, eagle, eagle3, or dflash. The older --speculative-model and --num-speculative-tokens flags are deprecated.

vllm serve <target-model> \
  --speculative-config '{
    "method": "ngram",
    "num_speculative_tokens": 4,
    "prompt_lookup_min": 2,
    "prompt_lookup_max": 5
  }'

vllm serve Qwen/Qwen3-4B-Thinking-2507 \
    --speculative-config '{"model": "Qwen/Qwen3-0.6B", "num_speculative_tokens": 5, "method": "draft_model"}'

from vllm import LLM

llm = LLM(
    model="meta-llama/Meta-Llama-3-8B-Instruct",
    tensor_parallel_size=2,
    speculative_config={
        "model": "RedHatAI/Llama-3.1-8B-Instruct-speculator.eagle3",
        "draft_tensor_parallel_size": 2,
        "num_speculative_tokens": 2,
        "method": "eagle3",
    },
)

SGLang

SGLang selects the variant with --speculative-algorithm: EAGLE (EAGLE-2), EAGLE3, NEXTN (MTP heads, an alias of EAGLE), STANDALONE (a separate draft model), NGRAM, or DFLASH. Model-based variants take --speculative-draft-model-path. Three flags shape the draft tree: --speculative-num-steps (drafting depth), --speculative-eagle-topk (branching per step), and --speculative-num-draft-tokens (how many candidates the target verifies).

python3 -m sglang.launch_server \
    --model meta-llama/Llama-2-7b-chat-hf \
    --speculative-algorithm EAGLE \
    --speculative-draft-model-path lmsys/sglang-EAGLE-llama2-chat-7B \
    --speculative-num-steps 3 \
    --speculative-eagle-topk 4 \
    --speculative-num-draft-tokens 16 \
    --mem-fraction-static 0.7 \
    --cuda-graph-max-bs-decode 8

When It Does Not Help

Speculative decoding adds draft work to every step. That work is repaid only when the target accepts enough tokens. Three conditions push α down far enough that the system can end up slower than plain decoding.

Batch size matters too. The memory-bound argument is strongest at low concurrency, where one forward pass serves one token. As the batch fills, the GPU becomes compute-bound and the spare capacity that verification borrows shrinks. vLLM's documentation frames speculative decoding as reducing inter-token latency under medium-to-low QPS, memory-bound workloads, and grades n-gram and suffix decoding as the methods that add the least load at peak traffic. EAGLE-3's 1.38x throughput gain at batch size 64 shows the effect does not vanish at scale, but it is smaller than the single-stream 6.5x.

For the other levers that stack with speculative decoding, see LLM inference optimization, continuous batching, and FP8 quantization. Quantization cuts the bytes moved per pass; speculation cuts the number of passes; batching fills the pass with more sequences. They multiply.

Frequently Asked Questions

What is speculative decoding?

An inference algorithm that samples from an autoregressive LLM faster without changing its outputs. A cheap draft proposes K candidate tokens. The target model verifies all K in a single forward pass, which on a memory-bound GPU costs about the same as generating one token. Every accepted token is nearly free. Leviathan et al. measured 2x-3x on T5-XXL with identical outputs.

Does speculative decoding change output quality?

No. Modified rejection sampling provably preserves the target's output distribution within hardware numerics. Leviathan et al. reported identical outputs on T5-XXL at 2x-3x; Chen et al. reported 2-2.5x on Chinchilla 70B without compromising sample quality. No retraining or architecture change to the target is needed.

How much speedup does speculative decoding give?

Theorem 3.8 of Leviathan et al.: (1 - α^γ+1) / ((1 - α)(γc + 1)). With a free draft, α = 0.8 and γ = 5 gives 3.69x; α = 0.9 and γ = 10 gives 6.86x. Published: 2x-3x on T5-XXL, 2-2.5x on Chinchilla 70B, 2.7x-3.5x for EAGLE on LLaMA2-Chat 70B, 3.05x-4.26x for EAGLE-2, up to 6.5x for EAGLE-3.

What is the difference between a draft model, Medusa, and EAGLE?

A draft model is a separate smaller model; no training, but a second model to serve. Medusa adds decoding heads to the target and verifies with tree attention (Medusa-1 over 2.2x, Medusa-2 2.3-3.6x). EAGLE trains a one-layer head that drafts from the target's internal features (2.7x-3.5x on LLaMA2-Chat 70B; EAGLE-2 3.05x-4.26x; EAGLE-3 up to 6.5x). The EAGLE README reports EAGLE-1 at 3x vanilla, 2x Lookahead, and 1.6x Medusa on a 13B model.

How do I enable speculative decoding in vLLM or SGLang?

vLLM: pass a JSON object to --speculative-config with method (draft_model, ngram, eagle, eagle3, mtp, suffix), an optional model, and num_speculative_tokens. SGLang: --speculative-algorithm (EAGLE, EAGLE3, NEXTN, STANDALONE, NGRAM) with --speculative-draft-model-path and the --speculative-num-steps, --speculative-eagle-topk, --speculative-num-draft-tokens tree flags. Full commands in the section above.

When does speculative decoding not help?

When α is low: a poor or out-of-distribution draft, high sampling temperature, or a draft that is not much cheaper than the target. Leviathan et al. measured 3.4x at temperature 0 versus 2.6x at temperature 1 on the same task. Hugging Face recommends an assistant at least an order of magnitude smaller than the target and input-grounded tasks.

How does Morph use speculative decoding?

morph-v3-fast merges code edits with n-gram prompt-lookup speculation over a 64-token draft window and serves at 10,500 tok/s. Morph's open coding models (Kimi K3, GLM-5.3, GLM-5.3-Flash, DeepSeek V4 Flash) run speculators trained on coding traffic; DeepSeek V4 Flash reaches up to 150 tok/s on private deployments with a custom speculator.

Related Resources

• How Does Speculative Decoding Work? The plain-language version.
• LLM Inference Optimization
• Continuous Batching for LLM Serving
• FP8 Quantization
• AI Inference
• LLM Inference: Cost, Speed, Caching, and GPU Capacity
• Fast Apply Model
• Dedicated Inference