Speculative Decoding

Speculative decoding is the technique of producing several candidate tokens cheaply and verifying them with a single batched forward pass of the main model. If the candidates are accepted, you generate multiple tokens for the cost of one forward pass; if they're rejected, you fall back to the standard sequential generation. The interesting variation is in how the candidates are produced — and as of 2026, llama.cpp supports four mechanisms, each with different memory/quality tradeoffs.

The arithmetic that makes it worthwhile: producing a draft token has to be substantially cheaper than running the main model, and the acceptance rate has to be high enough that the batch verification pays off. Both depend on the workload — heavily structured outputs (code, translation, math) get higher acceptance than creative writing.

The four llama.cpp variants

Draft model (--spec-type draft). Use a small sibling model from the same family — same tokenizer, same vocabulary, same general style — to generate draft tokens. The main model then verifies them in batch. The classical implementation. The ratio between the two models matters: the draft has to be at least ~3× smaller in active parameters or the overhead eats the speedup. Worked example from Vyacheslav: Gemma-4 31B UD-Q4_K_XL alone at 62 t/s; Gemma-4 31B with Gemma-4 E2B as draft, 97 t/s (~55% draft acceptance on coding workloads). Specify --parallel 1 to avoid the 4× memory reservation that automatic parallelism takes.

Default --draft-p-min is 0.75 — accept draft tokens only when both the draft and main model are >75% confident. Some output drift is technically allowed: a token can land that the main model wouldn't itself have picked but agrees is acceptable. Lossless verification is not the default for the draft-model path.

MTP — Multi-Token Prediction (--spec-type mtp). Extra prediction heads trained jointly with the model that predict several tokens ahead from the same hidden state. No separate draft model, no separate weights to load (the heads live inside the GGUF). First shipped in DeepSeek V3/R1; llama.cpp PR #22673 brought it to Qwen 3.6 and Gemma 4.

MTP at --spec-draft-n-max 2-3 is the typical configuration. janvitos's measurement on RTX 4070 Super 12GB (reddit-qwen3-6-mtp-12gb): 80+ t/s on code with 92–95% draft acceptance. The dependency on context-sensitive correctness is lower than a separate draft model because the heads were trained alongside the main weights. Default verification is exact — MTP doesn't allow the output drift the draft-model path does.

EAGLE3 (--spec-type eagle3). A lightweight autoregressive head bolted onto existing weights — no draft model, no jointly-trained MTP heads. Reads the hidden state of the main model and predicts the next several tokens from it. Works on models that weren't designed with speculative decoding in mind. llama.cpp PR #18039. Default verification is exact.

ngram-mod (--spec-type ngram-mod). Pure-math draft from prior tokens in the same context — no model, no head. Excellent when the output reproduces large chunks of the input verbatim: making a small edit to a file and emitting the rest unchanged, finding-and-replacing a variable name across a block, regenerating code with one change. llama.cpp PR #19164. This is the agentic-coding-friendly variant: when an agent loops "read file → edit → emit modified file," ngram-mod can carry most of the rewrite for free.

When each variant wins

The context-dependence

A finding from MTP measurements at long context (Still-Notice8155, reddit-qwen3-6-mtp-12gb): MTP saves roughly constant time per token regardless of context length, while attention cost grows linearly with context. So MTP's relative benefit widens as context fills up. On Qwen3.6-35B-A3B at 125K context: non-MTP ~3 t/s vs MTP+turboquant 13.6 t/s — 4.5× speedup. At 5K context the same setup is 1.8× faster. This is the opposite shape from many optimizations and worth keeping in mind: MTP's payoff is biggest exactly where unmodified inference is slowest.

Draft acceptance is workload-dependent

MTP on Qwen3.6 across nine task types (janvitos's mtp-bench.py):

The spread is the predictability of the output token distribution. Code with established patterns gets ~95% acceptance; open-ended prose drops to ~69%. The ranking is stable across the speculative-decoding family — workloads that are friendly to MTP are also friendly to draft-model and EAGLE3 variants.

Failures and caveats

• Vision is broken on MTP as of mid-2026. llama.cpp issues #22867 and PR #22673 thread document the current state. Stick to draft-model for multimodal.
• Memory for the draft model is not free. --parallel 1 halves the KV cache reservation that defaults to --parallel 4; the difference can be the GB that decides whether MTP fits on the GPU.
• Reasoning model output through speculative decoding needs --chat-template-kwargs '{"preserve_thinking": true}' so the thinking trace isn't truncated. The MTP heads in Qwen 3.6 were trained with this assumption.

Cross-references

• reddit-qwen3-6-mtp-12gb — practitioner Reddit thread, the source of the empirical numbers above
• habr-local-llm-quantization-deep-dive — Vyacheslav's primer that catalogs all four variants
• moe-cpu-offload — the other operational lever that combines with speculative decoding to make 12 GB consumer cards viable for 35B-class models
• local-llm-16gb-vram-tests — the tests article doesn't enable speculative decoding; speeds reported there are sequential
• llama-cpp — implementation site