Speculative Decoding: Draft-Target, Medusa, EAGLE, N-grams

A language model writes its answer one token at a time, and every token needs a full read of the model's weights from GPU memory. For a large model that read is the slow part. Writing 200 tokens means reading the weights 200 times, while the arithmetic units of the GPU sit mostly idle waiting for data. Speculative decoding uses that idle arithmetic. A cheap guesser proposes several next tokens, and the main model checks all of them in a single pass. Checked guesses that are right come out for free.

This lesson explains the mechanism with small calculations, then the four families of guessers: a second small model (draft-target), extra output heads (Medusa), a purpose-built small model that reads the main model's internal state (EAGLE), and a dictionary built from the text itself (n-gram and Lookahead Decoding). Afterwards you can predict whether speculation helps a given workload, estimate its speedup, and choose between the algorithms. For a longer walk-through of the verification and sampling logic, the Local LLMs course has Speculative Decoding. This lesson focuses on the production view.

Terms from earlier lessons. Decode is the phase after the prompt is read, where tokens are generated one after another (see How an LLM Generates Text). Memory-bound means speed is limited by memory bandwidth, not by arithmetic (Finding the Bottleneck). A forward pass is one trip of the data through all layers of the model. TTFT is time to first token. TPS is tokens per second, and ITL (inter-token latency) is the gap between two output tokens.

Example 1: the accept-and-extend mechanism

The decode phase is autoregressive: each token depends on the previous ones, so tokens are produced one at a time. The bottleneck is memory bandwidth. At low and moderate batch sizes (the number of requests processed together), compute sits idle while weights stream in from memory. If the engine could produce two, three, or more tokens for each trip of the weights through memory, it would produce far more tokens per second.

All speculation algorithms share the same three steps:

1. The speculator generates one or more draft tokens (guesses).
2. The target model (the model being accelerated) performs validation: it checks whether those tokens match what it would have generated itself.
3. The target model accepts the valid draft tokens and generates one more token of its own, which completes the forward pass.

graph LR
    A["Speculator proposes N draft tokens"] --> B["Target model validates all N in one forward pass"]
    B --> C["Accepts the first M valid tokens"]
    C --> D["Target adds 1 token of its own"]
    D --> E["M + 1 tokens produced"]

So a pass that accepts N draft tokens produces N + 1 tokens. Even if every guess is wrong, the pass still produces the one token plain decoding would have made. Speculation never makes the answer worse. It only makes the passes more productive.

Checking is cheaper than generating, and an analogy shows why: in a sudoku, solving is hard, while checking a finished grid is easy. For the target model, generating a token is like solving, and validating a draft token is like checking. The analogy has a limit: validation is not free. It still costs extra arithmetic, which is why the arithmetic must be idle for speculation to pay off. Generating the draft tokens also costs compute and memory.

Speculation improves only TPS and ITL. It does not help TTFT, because it acts on the decode phase, after the first token.

The average number of tokens per pass follows from a simple model. Suppose each draft token is accepted with probability a (the acceptance rate), independently, and k tokens are drafted. The chance that the first i tokens are all accepted is a to the power i. The expected number of accepted tokens is the sum of a^i for i from 1 to k, and one more token always comes from the target. A hand case with a = 0.8 and k = 3: 0.8 + 0.64 + 0.512 = 1.952 accepted, so 2.952 tokens per pass. The table below applies the same sum to several settings.

by-hand case a=0.8 k=3: 2.952
tokens per pass for acceptance rate a and draft length k
k     a=0.5  a=0.7  a=0.8  a=0.9  
1     1.50   1.70   1.80   1.90   
2     1.75   2.19   2.44   2.71   
3     1.88   2.53   2.95   3.44   
4     1.94   2.77   3.36   4.10   
5     1.97   2.94   3.69   4.69   
6     1.98   3.06   3.95   5.22   
7     1.99   3.14   4.16   5.70   
8     2.00   3.20   4.33   6.13

Look at the columns. With a = 0.5 the count flattens quickly near 2 tokens per pass, no matter how many tokens are drafted. With a = 0.9 it keeps growing. This shows why a high acceptance rate matters more than a long draft: guesses far down the list only count if every earlier guess was right.

Example 2: draft cost, draft length, and acceptance

The performance gain from any speculation method depends on three factors:

1. Draft token cost: the time it takes to generate one draft token.
2. Draft sequence length: how many draft tokens are generated per forward pass.
3. Token acceptance rate: the percentage of draft tokens the target accepts.

Acceptance is high early in the draft and drops deeper in the sequence, because each guess builds on the previous guess. This is also why one wrong token ends the useful part of the draft: once a draft token is rejected, every token after it is rejected too, since they were guessed on top of a wrong token.

The aim is therefore short drafts with a high acceptance percentage. Generating and validating tokens is cheap compared with generating them with the original model, but the overhead is still significant.

The next table turns this into a speedup estimate. It measures time in units of one target pass. A draft token costs c of a pass (for example 0.05). One round costs 1 + k x c, and it yields 1 + a + ... + a^k tokens. Speedup is tokens per round divided by cost per round, compared with plain decoding at 1 token per unit. A second table also uses an acceptance rate that falls with depth, which is closer to reality.

speedup versus plain decoding, a = 0.8
k    c=0.05         c=0.15        
1    1.71           1.57          
2    2.22           1.88          
3    2.57           2.04          
4    2.80           2.10          
5    2.95           2.11          
6    3.04           2.08          
7    3.08           2.03          
8    3.09           1.97          
9    3.08           1.90          
10   3.05           1.83          
best k: 8 for c=0.05, 5 for c=0.15

acceptance that falls with depth (a0 = 0.9, decay 0.85), c = 0.1
k=1 tokens/pass 1.90 speedup 1.73
k=2 tokens/pass 2.59 speedup 2.16
k=3 tokens/pass 3.04 speedup 2.34
k=4 tokens/pass 3.28 speedup 2.35
k=5 tokens/pass 3.40 speedup 2.27
k=6 tokens/pass 3.45 speedup 2.15
k=7 tokens/pass 3.46 speedup 2.04
k=8 tokens/pass 3.47 speedup 1.93

Reading the first table: a cheaper draft (c = 0.05) allows a longer useful draft (best k is 8, speedup 3.09) than an expensive one (c = 0.15, best k is 5, speedup 2.11). The best k is where one more guess adds fewer tokens than it adds cost. Past the best k the speedup falls, which matches the advice to keep drafts short. The second table shows the same effect from the other side: when later guesses are less reliable, the tokens per pass nearly stop growing after k = 5 (3.40, 3.45, 3.46, 3.47), but each extra draft token still costs time, so the speedup peaks at k = 4 (2.35) and then declines.

Real systems have more variables than these formulas, and three of them matter most:

• Temperature. A higher temperature makes the target's next-token probabilities flatter and harder to predict, which lowers the acceptance rate.
• Subject matter. A draft model or extra head that knows math better than history will be accepted more often on math.
• Batch size. Speculation works best at low batch sizes, where spare compute cycles exist. At higher batch sizes the compute is too saturated to afford verification, so speculation must be switched off dynamically.

Each algorithm below handles these trade-offs differently, and choosing the right one for the situation can give large TPS gains.

Example 3: draft-target speculative decoding

The original method uses two models:

• Draft model: an extra, smaller model that writes the guesses.
• Target model: the original model, which now also verifies the guesses while doing normal decode.

The most important choice is the draft model. A good one has a high acceptance rate and needs few resources. Draft models are usually smaller members of the same family as the target, because they share a tokenizer (the rule that cuts text into tokens) and similar behavior. As a rule of thumb, the draft model should have at least ten times fewer parameters than the target. Fine-tuning or distillation (training a small model to imitate a big one) can raise the acceptance rate by teaching the small model to behave more like the target.

Draft-target is a good choice when you want something that works out of the box with no training or fine-tuning. Other algorithms usually perform better, though, because draft-target adds the most overhead of any method:

• The engine must store the draft model's weights, activations, and KV cache in memory.
• It must spend compute on the draft model's own prefill.
• The two models must run in step so they do not compete for resources. Inference engines such as TensorRT-LLM handle this orchestration.

sequenceDiagram
    participant D as Draft model (small)
    participant T as Target model (large)
    D->>D: generate k draft tokens
    D->>T: send draft tokens
    T->>T: one forward pass validates all of them
    T-->>D: accepted prefix plus 1 target token

Example 4: Medusa

Medusa was one of the first alternatives to draft-target. It removes the cost of running a second model by changing the target itself. The target is fine-tuned so that one forward pass produces several tokens.

An ordinary LLM has one decoder head, the final layer that turns the model's internal state into a token. Medusa grafts two to four extra heads onto the model. Each extra head predicts one more token further ahead, so together they generate a sequence of draft tokens. As in draft-target, the draft tokens are validated on the next forward pass.

Medusa matters mostly because it inspired later techniques such as EAGLE.

Example 5: EAGLE

An off-the-shelf small model has a basic mismatch with the job. A model such as Qwen 0.5B is built to be a good stand-alone LLM on cheap hardware. It was not built to speculate tokens on a GPU such as a B200, so it is inefficient to run and has a relatively low acceptance rate.

EAGLE is a purpose-built draft model trained from scratch for speculation. It can propose up to eight draft tokens (twice as many as Medusa) with a very high acceptance rate.

The key idea is hidden states. While a big LLM predicts a token, its layers pass internal vectors to each other, called hidden states. They hold a lot of context about the predicted tokens. A traditional draft model cannot see them. EAGLE is trained to take hidden states as input and produce speculative tokens as output. It is trained on three of them: one from an early layer, one from a middle layer, and one from a late layer. EAGLE models are often under one billion parameters and improve when given more training data.

graph LR
    T["Target model layers"] -->|early, middle, late hidden states| E["EAGLE draft model"]
    E --> D["Up to 8 draft tokens"]
    D --> V["Target validates"]

In practice, with an engine such as TensorRT-LLM, implementations are straightforward: EAGLE is created after the target is trained (post-training) and speculates a single sequence. EAGLE can be attached to the same module (the same PyTorch class) as the target. Each forward pass then runs both the target and the EAGLE speculator. This solves the second problem of draft-target, where several round trips to the CPU were needed to coordinate two separate models.

EAGLE is the go-to speculation algorithm for general use, for engineers who can train EAGLE heads, and it is well supported by engines. Like every speculation technique, using it to lower latency requires smaller batch sizes. That lowers throughput and raises the cost per token.

Example 6: n-gram speculation and Lookahead Decoding

N-gram speculation has no draft model at all. An n-gram is a run of N tokens that appeared in order. While the engine builds the KV cache, it also builds an n-gram dictionary that maps one starting token to an observed sequence of N tokens that followed it.

The dictionary holds common sequences from the input text and is first built during prefill. During decode, the token just generated is looked up in the dictionary, and any stored continuation becomes the draft. On the next forward pass the target validates it as usual. This works especially well for constrained languages such as code, where the same patterns repeat.

The result below shows a small dictionary built from a code prompt (23 tokens such as for ( int i = 0 ; i < n ; i ++ ) { total += values [ i ] ; }, N = 4), then drafts from it. Each key is a single token, and its value is the run of tokens that followed it the first time it was seen.

entries: 16
draft after 'for'   : [(, int, i, =]
draft after 'total' : [+=, values, [, i]
last generated token: j, draft: []
unknown token 'while': []

The dictionary is cheap: no model runs, only table lookups. Notice what the lookups show. After for the draft is a long run of tokens taken from the prompt. The token j was never seen in the prompt, so it yields no draft at all, and while gets nothing either. This is the main limit: the acceptance rate is only high when the model's output is similar to its input.

Comparing n-grams with EAGLE:

Lookahead Decoding is a related method. Instead of reading n-grams from the input, it generates n-grams during inference to fill the dictionary. It is more general than n-gram speculation, because it does not depend as much on highly repetitive context. The cost is extra compute to generate the n-grams.

Every speculation algorithm has the same goal: fewer forward passes to produce a full answer, which improves overall latency and, specifically, tokens per second per user in decode. N-gram speculation excels at code completion and similar tasks, while Lookahead Decoding generalizes in systems that have excess compute.

Choosing an algorithm

For hardware and batching, see Autoscaling, Batching, Cold Starts, and Routing, and for where engines expose these settings, see Inference Engines.

Interview points

• Speculative decoding uses the idle compute of memory-bound decode to produce N + 1 tokens per forward pass, where N is the number of accepted draft tokens. It improves TPS and ITL, not TTFT.
• Three factors set the gain: draft token cost, draft length, and acceptance rate. Acceptance falls with depth, and one rejected token discards everything after it, so short high-acceptance drafts win.
• Higher temperature lowers acceptance. Larger batch sizes remove the spare compute, so speculation must be disabled dynamically at high batch sizes. Using speculation for latency lowers throughput and raises cost.
• Draft-target needs a draft model about ten times smaller from the same family. It is quick to set up but has the most overhead (weights, activations, KV cache, draft prefill, orchestration).
• Medusa adds two to four heads to the target through fine-tuning. EAGLE is a trained small model (often under 1B parameters) that reads early, middle, and late hidden states and drafts up to eight tokens. EAGLE is the general-purpose choice.
• N-gram speculation has no model and can draft over ten tokens, but only works when output resembles input, as in code editing. Lookahead Decoding fills the dictionary during inference and needs extra compute.

Summary

• Speculation turns idle decode compute into extra tokens: a speculator drafts, the target validates, and the target adds one token of its own.
• Expected tokens per pass grow with acceptance rate and draft length, with diminishing returns, and the speedup depends on the cost of the draft.
• Temperature, subject matter, and batch size change the acceptance rate and the benefit.
• Draft-target uses a second small model. Medusa adds heads. EAGLE uses a purpose-built model fed by hidden states. N-gram uses a dictionary of the input. Lookahead generates its n-grams during inference.
• Speculation trades throughput for latency, so it fits low-batch, latency-sensitive serving.