Speculative Decoding

Definition

Checking five guessed tokens costs a large language model almost exactly what it costs to write one — so you may as well guess. Speculative decoding is that observation turned into an algorithm. A small, cheap draft model writes the next few tokens; the large target model then scores all of them in a single forward pass and keeps the longest prefix it agrees with. Published results put it at 2x-3x faster on real workloads (Leviathan et al., 2023; Chen et al., 2023).

The part that makes it deployable rather than merely clever: the text it produces is drawn from exactly the same probability distribution as the target model would have produced alone. Not "almost the same". The same. Speculative decoding is not a quality-for-speed trade — there is no trade — which is why an inference stack can turn it on without anyone re-running an evaluation.

How It Works

The loop

The draft model generates the next γ tokens (γ = 4 or 5 in most deployments), one at a time — it is small, so this is quick. Those γ candidates are appended to the prompt and handed to the target model, which processes the whole extended sequence in one forward pass. That single pass yields the target's own probability distribution for every one of the γ positions in parallel, because each position's output depends only on the tokens before it.

The target then walks the candidates left to right, accepting or rejecting each by the rule below. It keeps the accepted prefix, replaces the first rejected token with one of its own, and the loop repeats. If every candidate is accepted, the pass has already computed the distribution for the position after the last one, so the target emits a free bonus token too — γ proposals can yield γ+1 tokens.

Why the verification pass is nearly free

This is the whole reason the technique works, and it is the cleanest demonstration of the memory wall anywhere in inference. A decode step is bandwidth-bound: to produce one token the GPU must read every weight in the model out of memory, and then does only about two floating-point operations per byte it read — hundreds of times below the ratio its tensor cores need to stay busy. The memory-wall page owns that argument; take it as given here.

Its consequence for speculative decoding is direct. You were going to read the whole weight matrix anyway. Having read it, doing five tokens' worth of arithmetic instead of one costs essentially nothing, because the arithmetic was never the bottleneck. Put numbers on it, using Llama 3.1 70B (70.6B parameters, model card) in FP8 on an H100 SXM (3.35 TB/s of bandwidth, 1,979 dense FP8 TFLOPS, NVIDIA):

1. Weights read per forward pass: 70.6 × 10⁹ × 1 byte = 70.6 GB → 70.6 ÷ 3,350 = 21.1 ms. This number does not change with how many tokens are in the pass.
2. Arithmetic for one token: 2 × 70.6 × 10⁹ = 141 GFLOP → 141 ÷ 1,979,000 = 0.07 ms.
3. Arithmetic to verify five candidates: 5 × 141 = 706 GFLOP → 0.36 ms.

The verification pass takes about 21.1 + 0.36 = 21.5 ms against 21.1 + 0.07 = 21.2 ms for an ordinary decode step — roughly 1.4% more time to do 5x the work. Generating those five tokens the normal way would have cost 5 × 21.2 = 106 ms. The spare capacity that speculative decoding spends is not a clever discovery; it is the 99%+ of the GPU that decoding leaves idle.

The acceptance rule, and why the output is identical

The draft model proposes token x with probability q(x); the target assigns it probability p(x). The rule is:

• Accept x with probability min(1, p(x) / q(x)). If the target likes the token at least as much as the drafter did, it is kept unconditionally.
• Otherwise reject it and resample from the residual distribution — p minus q, with negatives clipped to zero, renormalised — which puts the mass exactly where the drafter had been over-confident.

Leviathan et al. prove that a token produced this way is distributed identically to one sampled from p directly. The rejection branch is not damage control; it is what makes the guarantee hold. An implementation that simply resamples from p on rejection, or that skips the residual step, quietly biases the output — and will still look fine in a demo. (With greedy decoding, at temperature 0, the rule collapses to "accept while the draft token matches the target's argmax", and the output is literally token-for-token what you would have got.)

What the speedup actually is

Each token survives only if all the tokens before it did, so with a per-token acceptance probability α the expected number of tokens emitted per verification pass is a geometric sum (Leviathan et al., Theorem 3.8):

E[tokens] = (1 − α^(γ+1)) / (1 − α)

The cost of a cycle is one target pass plus γ draft passes. Writing c for the ratio of a draft pass to a target pass (measured at under 0.05 for their small drafters, and 0.11 for a drafter one-fourteenth the target's size), a cycle costs 1 + γc target passes, and:

speedup = (1 − α^(γ+1)) / [(1 − α)(1 + γc)]

Take α = 0.80 — the acceptance rate Leviathan et al. measured for a 250M-parameter T5-base drafting for the 11B T5-XXL — with γ = 5 and c = 0.05:

• Expected tokens: (1 − 0.8⁶) / 0.2 = 0.7379 / 0.2 = 3.69
• Cost: 1 + 5(0.05) = 1.25 target passes
• Speedup: 3.69 / 1.25 = 2.95x — which is where the "2x-3x" in the abstract comes from.

Now let α fall. At α = 0.4 the expected yield drops to (1 − 0.4⁶) / 0.6 = 1.66 tokens and the speedup to 1.33x. At α = 0.2 the yield is 1.25 tokens — exactly the 1.25 passes the cycle cost — and the technique breaks even: you are running two models to get the throughput of one. Below that it is a straight loss. With the heavier drafter (c = 0.11, cost 1.55 passes) break-even arrives much earlier, at α ≈ 0.36, which is the real argument for keeping the draft model tiny: a better drafter raises α, but it raises c faster.

There is also an optimal γ. Guessing further ahead can only add tokens with probability α^k, which decays, while the draft cost γc grows linearly — so with α = 0.8 and c = 0.05 the speedup peaks around γ = 8 (about 3.1x) and then declines. Speculating harder is not the same as speculating better.

Real-World Applications

vLLM ships it, and its own benchmarks show both faces of the technique. Serving Llama 3 70B on 4x H100s, vLLM measured up to 1.5x faster token generation with a draft model on ShareGPT, and up to 2.8x with n-gram drafting on CNN/DailyMail summarization — but at high query rates on the same hardware, a 1.4x slowdown with the draft model and a 1.8x slowdown with n-grams (vLLM blog, October 2024). One technique, one cluster, a 4x swing depending on load.

n-gram (prompt-lookup) drafting is the cheapest possible drafter: no model at all. The proposed tokens are copied from the prompt, by string match. It costs nothing (c ≈ 0) and it is useless on open-ended chat — but on summarization, code editing and retrieval-augmented answering, where long spans of the output are spans of the input, the acceptance rate is high enough to hit that 2.8x. The lesson generalises: the best drafter is whatever cheaply predicts this workload.

DeepMind's original deployment was at frontier scale. Chen et al. accelerated Chinchilla 70B by 2-2.5x in a distributed setup with no change to sample quality — and reported 1.92x on XSum summarization against 2.46x on HumanEval code, same models, same drafter. A 28% difference purely from what was being written.

Draft models are increasingly distilled from their targets. Imitating the target's distribution with a fraction of the parameters is the definition of knowledge distillation — which is why labs now train small models specifically as drafters rather than grabbing the smallest checkpoint they happen to own.

The lossy relaxation is a live research direction. Google's speculative cascades replace strict verification with a flexible deferral rule, accepting a good draft answer even when it is not the one the target would have chosen. That buys a higher acceptance rate and gives up the identical-distribution guarantee — the same shape of technique, with the trade made explicit.

Key Concepts

• Acceptance rate (α) is the whole ballgame. It is a property of the draft-target pair on a given workload, not of either model — the same drafter can sit at 0.8 on translation and collapse on a domain it has never seen. Every reported speedup is really a report of α.
• The cost coefficient (c) is the tax you pay for the guesses. It is why the drafter must be one or two orders of magnitude smaller than the target, and why "use a slightly smaller model of the same family" is usually a bad drafter.
• The bonus token is why the exponent in the formula is γ+1: a fully accepted draft of γ tokens yields γ+1, because the verification pass has already computed the distribution one position past the end.
• Every accepted token still costs KV cache. The verification pass writes KV-cache entries for all γ candidates, and the rejected ones have to be rolled back — an inference server doing speculative decoding is managing a cache that occasionally has to be un-written.

Challenges

Draft-target divergence turns the speedup into a tax. The arithmetic above is unforgiving: below roughly α = 0.2 (or α = 0.36 with a heavier drafter) you are paying for two models and getting less than one model's throughput. And α is not stable — it is measured on a workload, and production traffic is not that workload. A drafter tuned on English chat meets a user pasting Terraform, or a language it barely saw in pretraining, and the acceptance rate falls off a cliff that no amount of tuning γ recovers. The failure is silent: nothing breaks, the output stays correct, the tokens just get slower and the GPU bill goes up.

On a busy server the spare compute it exploits does not exist. Speculative decoding cashes in the idle arithmetic of a bandwidth-bound decode step — but batching cashes in the same idle arithmetic, and there is only one of it. A fleet running large batches has already spent that capacity and is drifting toward compute-bound; the wasted work of a rejected draft now displaces real work for other users. The SmartSpec authors measured the crossover on Llama-7B with a 160M drafter on one A100 at α = 0.7: beyond a request rate of 12/s proposing 5 tokens gives no gain at all, and beyond 16/s even proposing 3 tokens makes things worse (Liu et al., 2024). A technique that improves single-stream latency can reduce aggregate throughput, and the two are not the same objective.

The drafter is a tenant on the same GPU. Its weights sit in the same HBM, its forward passes consume the same bandwidth, and it must share the target's tokenizer — a drafter with a different vocabulary cannot propose tokens the target can score. Every one of those is a constraint on which model you are allowed to pick, and they rule out most of the obvious candidates.

"Identical output" is a statement about exact arithmetic, not about floating point. The verification pass computes logits at a different batch shape than an ordinary single-token step, and standard GPU kernels are not batch-invariant: the reduction order changes, the last bits of the logits change, and a near-tie in the argmax can flip (Thinking Machines, 2025). The distributional guarantee holds; bitwise reproducibility against a non-speculative baseline does not, and a team diffing outputs token-by-token to "verify losslessness" will find rare mismatches and misdiagnose the algorithm.

Future Trends

The separate draft model is being eliminated. Methods like Medusa and EAGLE bolt lightweight prediction heads onto the target model itself, drafting from its own hidden states — EAGLE-3 reports 3.0x-6.5x over ordinary autoregressive decoding (Li et al., 2025). No second checkpoint to host, no vocabulary mismatch, and a drafter that is in-distribution with its target by construction: c shrinks toward zero while α rises, which is exactly the direction the speedup formula rewards.

The second shift is that speculation is becoming load-aware. Since the technique wins at low load and loses at high load, a fixed γ is wrong most of the time. Systems like SmartSpec vary the proposal length per request — down to zero, i.e. no speculation at all — from the current batch and the observed acceptance rate. Expect γ to become a dial the server turns second by second rather than a number in a config file.

Code Example

The acceptance rule, with the residual resampling that the correctness proof depends on. p and q are the target's and drafter's probability vectors over the vocabulary at one position.

import torch

def verify(draft_tokens, q_probs, p_probs):
    """Return the tokens to emit. q_probs/p_probs: [gamma, vocab]."""
    accepted = []
    for i, tok in enumerate(draft_tokens):
        p, q = p_probs[i], q_probs[i]
        # Accept with probability min(1, p(tok) / q(tok)).
        if torch.rand(1).item() < min(1.0, (p[tok] / q[tok]).item()):
            accepted.append(tok)
            continue
        # Rejected: resample from the residual, NOT from p.
        residual = torch.clamp(p - q, min=0)
        residual = residual / residual.sum()
        accepted.append(torch.multinomial(residual, 1).item())
        return accepted                      # stop at the first rejection
    # All gamma accepted: the pass already scored one position further.
    bonus = torch.multinomial(p_probs[-1], 1).item()
    return accepted + [bonus]

Two lines carry the guarantee. torch.clamp(p - q, min=0) is the residual — resampling from p here instead would over-weight tokens the drafter already favoured, and bias the output. And return inside the loop is what enforces the prefix rule: once a token is rejected, everything after it was drafted on a false premise and must be thrown away, however good it looked.