Speculative Decoding: Speedup, Variants, and How to Enable It

TL;DR

• What it is. A draft proposes K tokens; the target verifies all K in one forward pass; modified rejection sampling keeps the target's exact output distribution.
• Why it works. Decoding is memory-bandwidth-bound. Verifying K tokens reads the weights once, the same as generating one token.
• How fast. Speedup = (1 - α^γ+1) / ((1 - α)(γc + 1)). At acceptance rate α = 0.8 with a free 5-token draft, 3.69x. Published: 2x-3x (T5-XXL), 2-2.5x (Chinchilla 70B), 2.7x-3.5x (EAGLE, LLaMA2-Chat 70B), 3.05x-4.26x (EAGLE-2), up to 6.5x (EAGLE-3).
• The variants. Separate draft model, n-gram prompt lookup, Medusa heads, EAGLE feature-level drafts, and native multi-token-prediction heads. They differ in where the draft comes from and what it costs to run.
• How to turn it on. vLLM: a JSON --speculative-config. SGLang: --speculative-algorithm plus draft-tree flags. Both below.
• What the papers leave out. Lossless in distribution is not byte-identical output. Acceptance rate is not comparable across engines. Independent benchmarking measures 1.96x at batch size 1 falling to 1.21x at batch 128, and published sweeps include settings that run slower than no speculation at all. Reported failure modes below, each with a source link.
• In production. morph-v3-fast runs n-gram speculation with a 64-token draft window at 10,500 tok/s. Morph's open coding models run speculators trained on coding traffic.

New to the idea? How does speculative decoding work? is the plain-language version of this page.

Why Decoding Is Memory-Bound

Autoregressive generation produces one token per forward pass. Each pass loads every layer's weights from GPU memory into the compute cores, then does a small amount of arithmetic on a single token. At batch size one, the weight load dominates. The GPU's math units sit mostly idle while bytes stream in. Decoding is memory-bandwidth-bound, not compute-bound.

The spare compute is the opportunity. Running K candidate tokens through the model in one pass reads the weights once and does K tokens of arithmetic. On a memory-bound GPU that costs almost the same as one token. Leviathan et al. state it directly: parallel scoring of short continuations from a fast draft has latency comparable to sampling a single token from the target.

The Draft-Then-Verify Loop

One round does four things. The draft proposes γ candidate tokens autoregressively, which is cheap because the draft is small or model-free. The target runs one forward pass over the context plus all γ candidates and records its own distribution at each position. Verification compares candidates left to right and accepts each one with probability min(1, p(x)/q(x)), where p is the target and q is the draft. The first rejection stops the run; the target samples a corrected token from the residual distribution, and the next round starts there.

Concretely: the draft proposes 5 tokens, the target accepts the first 3 and rejects the 4th. Those 3 tokens plus the corrected 4th cost one target forward pass instead of four. Across many rounds, the average number of tokens produced per pass is the speedup, minus the draft's own cost.

// gamma = draft length, e.g. 5
const draft = draftModel.propose(context, gamma)          // cheap

// ONE target forward pass over context + all gamma candidates
const targetProbs = targetModel.forward([...context, ...draft])

// Accept left to right with prob min(1, p(x)/q(x)); stop at first reject.
// On reject, sample from norm(max(0, p - q)) so the output distribution
// is exactly the target's (Leviathan et al. 2022, Appendix A.1).
const { accepted, corrected } = speculativeSample(draft, targetProbs)

context.push(...accepted, corrected)   // accepted tokens were ~free

Why It Is Lossless

The accept/reject rule is not a heuristic. Leviathan et al. prove that sampling x from the draft q, keeping it when q(x) ≤ p(x), rejecting it with probability 1 - p(x)/q(x) otherwise, and resampling rejections from norm(max(0, p(x) - q(x))) yields samples distributed exactly as p(x). Chen et al. give the same proof independently. The target model's output distribution is preserved within hardware numerics. Greedy decoding with a greedy draft reduces to exact token matching.

This is why speculative decoding is safe to ship without an eval cycle. Quantization, distillation, and pruning each trade some quality for speed. Speculative decoding trades none. It requires no retraining and no architecture change to the target. Leviathan et al. reported identical outputs on T5-XXL at 2x-3x. Chen et al. reported 2-2.5x on Chinchilla 70B on XSum and HumanEval without compromising sample quality.

Lossless in distribution is not byte-identical in practice

The proof assumes both paths compute the same target probabilities. A real serving stack does not. vLLM's own documentation splits the guarantee into three layers: theoretical losslessness holds up to the precision limits of hardware numerics, algorithmic losslessness is checked by convergence tests on the rejection sampler, and output stability is not promised. Only the first two are covered by the paper and the engine's tests.

A NeurIPS 2026 Education Track tutorial by Lily Zhang and Madison Kanna measured the gap. Serving the same Qwen3-8B twice under greedy decoding, once vanilla and once with a DSpark draft, the two HTML outputs were identical for 8,299 characters, 76% of the page, then split at a single near-tie token where transition: color 0.2s became 0.3s. Final lengths were 10,924 and 10,535 characters. Their explanation: the speculative path runs different kernels, the logits shift within floating-point precision, and a near-tie falls the other way. A change in batch size alone can do the same thing.

Quantizing the target widens it. A llama.cpp report tested the same prompt across a matrix of configurations: on a Q4_K_M target, a model-based draft diverged from vanilla greedy output while n-gram speculation on the identical weights matched, and on a bf16 target the same draft matched vanilla in four of four runs (ggml-org/llama.cpp #25618). The distribution is still the target's. The exact byte stream is not guaranteed to be.

Speedup by Acceptance Rate

Two numbers set the speedup. The acceptance rate α is the probability the target accepts a draft token. The draft length γ is how many tokens are proposed per round. A third, the cost coefficient c, is the wall time of one draft step divided by one target step. Leviathan et al.'s Theorem 3.8 gives the expected wall-time improvement:

speedup = (1 - α^γ+1) / ((1 - α)(γc + 1))

Two consequences follow. The ceiling for any draft is 1/(1 - α): at α = 0.9 nothing gets past 10x no matter how long the draft. And γ has an optimum: past it, the extra draft tokens are mostly rejected while their cost γc keeps growing. The table evaluates the formula at c = 0.05, the highest value Leviathan et al. observed with a draft two orders of magnitude smaller than the target, and at c = 0 for a model-free draft such as n-gram lookup.

The paper's own Table 1 checks out against the formula: α = 0.8 with γ = 5 gives 3.69x, and α = 0.9 with γ = 10 gives 6.86x, both at c = 0. Its measured T5-XXL results land inside the table's middle rows: 2.6x at temperature 1 and 3.4x at temperature 0 on English-German translation, 2.3x and 3.1x on summarization. Temperature matters because a hotter target is less predictable, which lowers α.

The sliders are the formula, not a marketing curve. Two things are easier to see by moving them than by reading a table. Raising γ helps until the bar chart turns over, and where it turns depends on both α and c. And α, not γ, sets the ceiling: at α = 0.7 no draft length reaches 4x, because 1/(1 - α) is 3.3x.

The same curve, measured

vLLM published a proposal-length sweep on AMD MI300X and MI355X in August 2026 that reproduces the shape end to end. On Qwen3-8B with an EAGLE-3 draft on GSM8K, the throughput ratio climbs, peaks, and falls, and the first four settings are all slower than not speculating at all. Per-position acceptance is why: the first draft position is accepted 86% of the time and the seventh only 19%, so every token past the peak adds cost without adding output.

The same sweep on MATH500, the harder workload for the same model and draft, never gets above the baseline at all: 0.44x at one draft token, still climbing but under 1x by seven. Acceptance there is 89.0% at the first position, higher than the GSM8K run that did succeed. A high acceptance rate is not sufficient. If the draft costs too much per step, c dominates the denominator and the speedup falls below 1 no matter how well the draft predicts.

What Acceptance Rate Actually Measures

Acceptance rate is the number every tuning guide tells you to watch, and it is the number most often misread. Three things practitioners have measured are worth knowing before you compare yours to anyone else's.

It is not comparable across serving frameworks

A vLLM issue benchmarked the same target, the same draft, the same dataset, and the same decoding settings on two stacks and got different acceptance numbers in both directions. On Qwen3-32B with an EAGLE-3 head, GSM8K, three speculative tokens, greedy, vLLM reported 55.08% acceptance and a mean accepted length of 2.65 while SpecForge reported about 44% and 2.32. With a Qwen3-0.6B standalone draft on the identical setup, the order flipped: vLLM 66.57% and 3.00, SpecForge about 72% and 3.17 (vllm-project/vllm #42508). The issue is open with no maintainer explanation. Treat a published acceptance rate as a number from one engine's counter, not a property of the draft.

A rate pinned at 1/(γ+1) means the draft is broken, not weak

An SGLang report on NVIDIA-Nemotron-3-Super-120B-A12B running --speculative-algorithm NEXTN --speculative-num-steps 2 saw an acceptance rate of exactly 0.33 on every step: the base token accepted, both draft tokens rejected, forever. Throughput dropped from 14.4 to 10.9 tok/s, which is speculation running as pure overhead. The same weights reached 63% acceptance on vLLM, so the draft was fine and the integration was not; it was traced to a MoE activation the runner did not support (sgl-project/sglang #21138). A deterministic 1/(γ+1) is the signature to alert on.

Mean accepted length can rise while the speedup falls

The two numbers engines report, acceptance rate and mean accepted length, can move in opposite directions, and only one of them tracks throughput. A Berkeley benchmark of five speculative methods on vLLM and SGLang compared chain-style drafting against wider draft trees on Qwen3-8B and GSM8K. Widening from a chain of 3 tokens to a tree of 21 raised mean accepted length from 2.25 to 2.92, and dropped the acceptance rate from 0.415 to 0.095. At batch size 1 the tree was ahead, 1.85x against 1.65x. By batch size 64 the 21-token tree fell below 1x on every workload, because the target was verifying branches that were going to be thrown away. Their conclusion: chain-style verification is the more robust choice (SpecDecode-Bench).

The same study measured where the time actually goes. Verification by the target model takes 42% to 95% of total execution time depending on the configuration, and the rejection sampling that makes the method lossless takes under 1.7% in every setting they ran. Losslessness is not what costs you. Rejected tokens are: every one of them consumed a slot in the target's forward pass and produced nothing.

High acceptance does not mean good output

Acceptance measures how often the draft agrees with the target. It says nothing about whether the answer is any good. The NeurIPS 2026 Education Track tutorial ran a Qwen3-8B with a DFlash draft across five domains and found acceptance length falling from 5.24 on coding to 1.84 on creative writing, with frontend design holding high acceptance while the generated pages broke. Task scores moved independently: frontend design 54.5 to 45.5, creative writing 70 to 30, guardrail accuracy flat at 80, and a long-horizon agent benchmark rising from 20.0 to 27.5 because one flipped token led the accelerated run down a different tool-call path. The same authors note that the nine benchmarks EAGLE-3, DFlash, and DSpark report on cover 17% of OpenRouter's real token traffic.

What the Draft Is Changes the Speedup

The formula makes α the lever, and α is a property of the draft-target pair on a given workload. A generic small model from the same family agrees with the target often enough for 2x-3x. A draft trained on the target's own traffic agrees more often and pushes past 3x. A draft that can read the answer out of the prompt, as in code editing, agrees most often of all.

Leviathan et al. tested the extreme case: a trivial bigram model as the draft for T5-XXL gave α ≈ 0.2 on translation, enough for 1.25x at γ = 3 because the draft cost nothing. That observation is what became n-gram and prompt-lookup speculation. When the output copies long spans of the input, a table lookup drafts them perfectly, and the same free draft that gave 1.25x on translation gives far more on code editing. That is the regime Morph's Fast Apply model lives in.

The Variants: Draft Model, N-gram, Medusa, EAGLE, MTP

Every variant shares the verification step. They differ in where the draft comes from, whether it needs training, and what it costs per round.

Self-speculation is the variant most guides leave out, and it is the one that costs nothing to serve. There is no second model and no extra head: the target drafts with a subset of its own layers, then verifies with all of them. Ryoma Sato's survey of the field collects the reported figures, including Draft & Verify at up to 1.99x, LayerSkip at up to 2.16x, and DistillSpec improving on a standard draft by 10% to 45% by distilling the draft against the target instead of training it on raw text. The distillation result is the general lesson: what raises α is training the draft on the distribution the target actually produces.

Medusa (Cai et al., arXiv 2401.10774) keeps a single model. Extra decoding heads predict several future positions at once, and tree attention verifies many candidate continuations in one pass. Medusa-1 trains only the heads on a frozen target and exceeds 2.2x. Medusa-2 fine-tunes heads and target together for 2.3-3.6x.

EAGLE (Li et al., arXiv 2401.15077) moves drafting to the feature level. A small head autoregresses on the target's second-to-top-layer features, conditioned on the token sequence shifted by one step to resolve feature-level uncertainty. On LLaMA2-Chat 70B it reaches 2.7x-3.5x latency speedup and doubles throughput while preserving the output distribution. EAGLE-2 (arXiv 2406.16858) replaces the static draft tree with a context-aware dynamic one, using the draft head's calibrated confidence to decide where to branch, for 3.05x-4.26x, 20%-40% over EAGLE-1. EAGLE-3 (arXiv 2503.01840) drops feature prediction for direct token prediction with multi-layer feature fusion, trained with a technique the authors call training-time test, for up to 6.5x, about 1.4x over EAGLE-2, and a 1.38x throughput gain at batch size 64 inside SGLang.

DSpark (DeepSeek and Peking University, 2026) is the most recent shape of the idea and the one vLLM's adaptive-verification path currently targets. A parallel backbone drafts a run of tokens with confidence scores, a hardware-aware scheduler keeps the prefix likely to survive, and the target verifies only that prefix. The loop is unchanged; the scheduler decides how much of it to run. Offline it improves accepted length by 16% to 31% over the previous best drafters, and in DeepSeek's own V4 serving stack it accelerated per-user generation by 60% to 85% at matched throughput against their MTP-1 production baseline.

Speculative speculative decoding (Kumar, Dao and May, arXiv 2603.03251) attacks a different bottleneck: drafting and verification are serial, so the draft sits idle during every verify. Their Saguaro algorithm has the draft predict the verification result and start the next run before it lands, reporting on average 30% faster than optimized speculative decoding baselines and up to 5x over autoregressive decoding on open-source engines.

Published Speedups

Every number below is from the cited paper, README, or serving-framework documentation. Speedup ratios are relative to vanilla autoregressive decoding of the same target model on the same hardware.

The last two rows are the ones to plan against. Every result above them was measured in the configuration that shows the method at its best, usually a single stream. The independent numbers were measured across a sweep, and they include the settings where speculation loses.

The SGLang row is the cleanest apples-to-apples measurement: same model, same GPU, same benchmark, three configurations. EAGLE-3 gives 2.36x over the baseline on that setup, which sits at the α ≈ 0.7-0.8 rows of the acceptance-rate table above.

Speculative Decoding in Production at Morph

Morph serves inference for AI coding agents, and code is the workload where speculative decoding pays off most. Two production uses illustrate the two ends of the variant table.

Fast Apply, n-gram speculation. morph-v3-fast merges an edit snippet into a file and returns the full merged file. Most output tokens already exist in the input, so a prompt-lookup draft that copies the next 64 tokens from the original file is right almost every time it fires. That is the c = 0 column of the acceptance-rate table with a high α. Combined with continuous batching and custom kernels, morph-v3-fast serves at 10,500 tok/s. The same family of tricks compresses context with Compact at 33,000 tok/s.

The independent benchmark above measured exactly when that choice is right. SpecDecode-Bench correlated speedup against BLEU-4 overlap between prompt and output, a measure of how much of the output already appears in the input. Once that overlap passes about 0.6, meaning roughly 60% of the output's 4-grams are already in the prompt, n-gram speculation beat EAGLE and EAGLE-3 at every batch size they tested, by up to 100%. Their stated takeaway: for workloads with heavy prompt-output overlap, code editing among them, n-gram is the method to use, and it needs no training. Their oracle analysis puts the ceiling for adaptively combining n-gram with EAGLE on a code-editing benchmark at 4.9x.

Open coding models, trained speculators. General chat traffic has no input to copy from, so the open models Morph serves (Kimi K3, GLM-5.3, GLM-5.3-Flash, DeepSeek V4 Flash) run with draft heads trained on coding-agent traffic. A speculator trained on the same distribution it will see in production has a higher α than a generic one, which is the "draft tuned to your workload" bar in the comparison above. On private deployments with a custom speculator, DeepSeek V4 Flash reaches up to 150 tok/s per user.

Dedicated deployments train the speculator on your traffic. See dedicated inference and the benchmark evidence behind the per-user speed numbers.

How to Enable It: vLLM and SGLang

Both major open-source serving stacks ship speculative decoding behind launch flags. The flags below are taken from the current vLLM and SGLang documentation as of 2026-09-07.

vLLM

vLLM takes a single JSON object via --speculative-config (or speculative_config= on LLM(...)). The method key selects the variant: draft_model, ngram, suffix, mtp, eagle, eagle3, or dflash. The older --speculative-model and --num-speculative-tokens flags are deprecated.

vllm serve <target-model> \
  --speculative-config '{
    "method": "ngram",
    "num_speculative_tokens": 4,
    "prompt_lookup_min": 2,
    "prompt_lookup_max": 5
  }'

vllm serve Qwen/Qwen3-4B-Thinking-2507 \
    --speculative-config '{"model": "Qwen/Qwen3-0.6B", "num_speculative_tokens": 5, "method": "draft_model"}'

from vllm import LLM

llm = LLM(
    model="meta-llama/Meta-Llama-3-8B-Instruct",
    tensor_parallel_size=2,
    speculative_config={
        "model": "RedHatAI/Llama-3.1-8B-Instruct-speculator.eagle3",
        "draft_tensor_parallel_size": 2,
        "num_speculative_tokens": 2,
        "method": "eagle3",
    },
)

SGLang

SGLang selects the variant with --speculative-algorithm: EAGLE (EAGLE-2), EAGLE3, NEXTN (MTP heads, an alias of EAGLE), STANDALONE (a separate draft model), NGRAM, or DFLASH. Model-based variants take --speculative-draft-model-path. Three flags shape the draft tree: --speculative-num-steps (drafting depth), --speculative-eagle-topk (branching per step), and --speculative-num-draft-tokens (how many candidates the target verifies).

python3 -m sglang.launch_server \
    --model meta-llama/Llama-2-7b-chat-hf \
    --speculative-algorithm EAGLE \
    --speculative-draft-model-path lmsys/sglang-EAGLE-llama2-chat-7B \
    --speculative-num-steps 3 \
    --speculative-eagle-topk 4 \
    --speculative-num-draft-tokens 16 \
    --mem-fraction-static 0.7 \
    --cuda-graph-max-bs-decode 8

Production Gotchas Practitioners Report

The papers report speedups on clean benchmarks. The issue trackers of vLLM, SGLang, and llama.cpp report what happens when the same methods meet chat templates, quantization, and real context lengths. These four come up repeatedly and none of them appear in the vendor documentation.

Chat templates and frozen MTP heads

A model's built-in multi-token-prediction head is often trained during pretraining and then frozen through supervised fine-tuning and RLHF. The main model learns the chat format. The MTP head never sees it. Role markers and thinking tags become high-entropy positions the draft has no signal for, and acceptance falls: roughly 83% at one draft token on chat-formatted data against about 90% reported on pretraining-like text (ROCm/ATOM #609). The analysis names three fixes: retrain the MTP head on chat-formatted data, cut the draft length to one token when serving through a template, or lower the draft length adaptively near structural boundaries.

The context size you picked can be the bug

One report on a Qwen3.6-35B-A3B quantization found MTP acceptance dropping to 16% with a 1.12x speedup at --ctx-size 12032, then recovering to 71% and 1.91x at 12288. The collapse recurred near 2048-aligned values, including one context length where acceptance hit zero, and was traced to how the draft context shares KV-cache slots with the target (ggml-org/llama.cpp #23658). If acceptance looks wrong, sweep the context size before you blame the draft.

Relaxed acceptance thresholds are not lossless

SGLang exposes --speculative-accept-threshold-single, shipped at 1.0, where verification matches the target exactly (measured in the NeurIPS tutorial). Lowering it accepts draft tokens the rejection rule would have thrown out, which buys speed by giving up the guarantee. Benchmarking 1.0 down to 0.4 on a frontend-design task at temperature 1 accepted visibly more drift. At temperature 0 the threshold changed nothing and every setting returned byte-identical pages. If you have relaxed this flag, none of the losslessness claims on this page apply to your deployment.

When It Does Not Help

Speculative decoding adds draft work to every step. That work is repaid only when the target accepts enough tokens. Three conditions push α down far enough that the system can end up slower than plain decoding.

Batch size matters too, and this is where the published numbers and the deployed numbers diverge most. The memory-bound argument is strongest at low concurrency, where one forward pass serves one token. As the batch fills, the GPU becomes compute-bound and the spare capacity that verification borrows shrinks. vLLM's documentation frames speculative decoding as reducing inter-token latency under medium-to-low QPS, memory-bound workloads, and grades n-gram and suffix decoding as the methods that add the least load at peak traffic.

The most complete public measurement of the crossover is SpecDecode-Bench, a systematic evaluation of five speculative methods on vLLM across four models and batch sizes from 1 to 128 on H100s, by Liu, Yu, Park, Stoica and Cheung. Their summary is worth quoting in full: speculation always helps, but expect a lower speedup in realistic batch settings, not the 3x-4x numbers often cited at batch size 1. EAGLE reached 1.96x on Llama-3-70B at batch size 1 and 1.21x on Llama-3.1-8B at batch size 128. The degradation scales with model size: going from batch 1 to batch 32 on ShareGPT cost 4.3% of the speedup on the 8B model and 14.0% on the 70B, which is already compute-bound on four GPUs at small batch.

Two production reports put the same number on it from different directions. IBM, running speculators in an internal production environment with thousands of daily users, reported roughly 2x on 7B-13B language models and 3x on their 20B code model, and stated plainly that they begin to observe throughput reduction beyond a batch size of 64. A vLLM issue measured the far side of that: at client concurrency 256, dynamic speculative decoding cost a 12% to 25% throughput penalty against no speculation and inflated median time to first token roughly 6x, and the penalty persisted even with a configuration that produced almost no draft tokens, which places the cost in the speculative path itself rather than in wasted drafts (vllm-project/vllm #48494). EAGLE-3's 1.38x throughput gain at batch size 64 shows the effect does not vanish at scale, but it is far smaller than the single-stream 6.5x.

Expect to reproduce a fraction of any headline number on your own setup. One reporter running Llama-3-70B at tensor parallel 4 with a 0.5B draft and four speculative tokens measured a maximum of 1.4x, at batch size 1, against the 2.8x in the vendor blog post they were following. The issue was closed as stale with no maintainer reply (vllm-project/vllm #10318). Published speedups are measured at the concurrency, sequence length, and draft pairing that flatter them.

On a mixture-of-experts model, batch size 1 is the worst case, not the best

The classic advice, speculate at low concurrency because that is where the GPU is idle, was derived on dense transformers. Fergus Finn worked the roofline for a modern MoE model and found the intuition partly inverted. In a dense feedforward layer, a speculated token rides weights that are already loaded, so it is close to free. In an MoE layer, extra tokens route to experts that are not resident yet, so they pay to bring those experts in. Modeling DeepSeek-V4-Flash, he puts the marginal cost of a speculated token at roughly 0.85 of full price at batch size 1, falling toward free only once the batch passes the point where the experts are already resident, around 43 tokens for that routing configuration.

Two consequences follow, and they cut in opposite directions from the dense case. There is a low-batch region where the optimal draft length is zero and you are better off not speculating at all. And the memory-bound stretch where speculated tokens are roughly free is wider for an MoE model than a dense one, so the useful batch range sits higher than the classic advice suggests. Compressed attention narrows it again from the other side: with multi-head latent attention, the first speculated token can already push the attention kernel compute-bound. If you serve MoE models, the batch size at which speculation earns its keep is a number to measure on your own stack, not to inherit from a paper on Llama.

For the other levers that stack with speculative decoding, see LLM inference optimization, continuous batching, and FP8 quantization. Quantization cuts the bytes moved per pass; speculation cuts the number of passes; batching fills the pass with more sequences. They multiply.

Frequently Asked Questions

What is speculative decoding?

An inference algorithm that samples from an autoregressive LLM faster without changing its outputs. A cheap draft proposes K candidate tokens. The target model verifies all K in a single forward pass, which on a memory-bound GPU costs about the same as generating one token. Every accepted token is nearly free. Leviathan et al. measured 2x-3x on T5-XXL with identical outputs.

Does speculative decoding change output quality?

No. Modified rejection sampling provably preserves the target's output distribution within hardware numerics. Leviathan et al. reported identical outputs on T5-XXL at 2x-3x; Chen et al. reported 2-2.5x on Chinchilla 70B without compromising sample quality. No retraining or architecture change to the target is needed.

How much speedup does speculative decoding give?

Theorem 3.8 of Leviathan et al.: (1 - α^γ+1) / ((1 - α)(γc + 1)). With a free draft, α = 0.8 and γ = 5 gives 3.69x; α = 0.9 and γ = 10 gives 6.86x. Published: 2x-3x on T5-XXL, 2-2.5x on Chinchilla 70B, 2.7x-3.5x for EAGLE on LLaMA2-Chat 70B, 3.05x-4.26x for EAGLE-2, up to 6.5x for EAGLE-3.

What is the difference between a draft model, Medusa, and EAGLE?

A draft model is a separate smaller model; no training, but a second model to serve. Medusa adds decoding heads to the target and verifies with tree attention (Medusa-1 over 2.2x, Medusa-2 2.3-3.6x). EAGLE trains a one-layer head that drafts from the target's internal features (2.7x-3.5x on LLaMA2-Chat 70B; EAGLE-2 3.05x-4.26x; EAGLE-3 up to 6.5x). The EAGLE README reports EAGLE-1 at 3x vanilla, 2x Lookahead, and 1.6x Medusa on a 13B model.

What is acceptance rate in speculative decoding?

The fraction of draft tokens the target keeps during verification. It is the lever that sets the ceiling: no draft length beats 1/(1 - α), so a draft accepted 70% of the time tops out at 3.3x. Engines usually report the related mean accepted length, the tokens produced per target forward pass. Two cautions. The number is not comparable across frameworks, and high acceptance means the draft agrees with the target, not that the output is good.

What is MTP speculative decoding?

Multi-token prediction. Prediction heads trained into the base model itself, as in the DeepSeek V3 line, so the checkpoint ships with its own draft and there is no second model to load. vLLM exposes it as method: mtp and SGLang as --speculative-algorithm NEXTN. The common gotcha: an MTP head is often frozen after pretraining, so it never learns the chat template, and serving through one lowers acceptance to about 83% at one draft token against roughly 90% on pretraining-like text.

Is speculative decoding really lossless?

In distribution, yes. Byte for byte, not necessarily. vLLM's documentation states that output stability is not promised and that theoretical losslessness holds only up to hardware numerics. Measured: the same Qwen3-8B served greedily with and without a draft produced identical text for 8,299 characters, then split on one near-tie token, because the speculative path runs different kernels. Quantized targets widen the effect. Relaxing an engine's acceptance threshold below 1.0 gives up the guarantee outright.

How do I enable speculative decoding in vLLM or SGLang?

vLLM: pass a JSON object to --speculative-config with method (draft_model, ngram, eagle, eagle3, mtp, suffix), an optional model, and num_speculative_tokens. SGLang: --speculative-algorithm (EAGLE, EAGLE3, NEXTN, STANDALONE, NGRAM) with --speculative-draft-model-path and the --speculative-num-steps, --speculative-eagle-topk, --speculative-num-draft-tokens tree flags. Full commands in the section above.

Does speculative decoding still help at high batch size?

Less, and past a point not at all. Speculation borrows the idle compute that exists while decoding is memory-bandwidth-bound, and a full batch removes it. Independent benchmarking measured EAGLE at 1.96x on Llama-3-70B at batch size 1 falling to 1.21x at batch size 128, with a 21-token draft tree dropping below 1x on every workload by batch 64. Mixture-of-experts targets invert part of this: there a speculated token costs the most at batch size 1, because it routes to experts that are not resident yet.

When does speculative decoding not help?

When α is low: a poor or out-of-distribution draft, high sampling temperature, or a draft that is not much cheaper than the target. Leviathan et al. measured 3.4x at temperature 0 versus 2.6x at temperature 1 on the same task. Hugging Face recommends an assistant at least an order of magnitude smaller than the target and input-grounded tasks.

How does Morph use speculative decoding?

morph-v3-fast merges code edits with n-gram prompt-lookup speculation over a 64-token draft window and serves at 10,500 tok/s. Morph's open coding models (Kimi K3, GLM-5.3, GLM-5.3-Flash, DeepSeek V4 Flash) run speculators trained on coding traffic; DeepSeek V4 Flash reaches up to 150 tok/s on private deployments with a custom speculator.

Related Resources

• How Does Speculative Decoding Work? The plain-language version.
• LLM Inference Optimization
• Continuous Batching for LLM Serving
• FP8 Quantization
• AI Inference
• LLM Inference: Cost, Speed, Caching, and GPU Capacity
• Fast Apply Model
• Dedicated Inference