Adaptive Speculation

The problem with a fixed draft length

Speculative decoding drafts several tokens ahead with a small model, then lets the target model verify them in one pass. Accepted tokens are free. The usual implementation drafts a fixed number of tokens every step. When the draft model's guesses are good, that's efficient. When acceptance drops, a fixed draft length keeps paying for tokens that get thrown away, and throughput collapses.

That failure mode is common with newer techniques like DFlash2 and multi-token prediction (MTP), where acceptance rates swing hard depending on context: exactly the case a fixed n can't adapt to.

Adaptive draft length

--spec-draft-adaptive is a patch to llama.cpp that scales draft length to the model's measured acceptance rate instead of a fixed number: draft length follows measured acceptance instead of a fixed n. When acceptance is high, it drafts further ahead; when it drops, it pulls back, rather than continuing to pay for rejected tokens.

We tested it on DFlash2, a draft model for Qwen3.8-27B, a new open-weight model roughly on par with Claude Opus 4.6 in quality: the kind of model where running it fast locally, on a laptop-class APU rather than a rack of GPUs, actually matters.

Two ways to draft: a sidecar model and a joint head

DFlash2 is a separate draft model: a small model, quantized and shipped on its own, that stands in front of a same-family target. MTP (multi-token prediction) is different: it's a head trained jointly with the target model itself, so it drafts from the target's own representations rather than approximating them from outside. Both plug into --spec-draft-adaptive the same way; how well adaptive drafting pays off depends on what a drafted token costs.

For DFlash2's cheap sidecar, that cost is low, so drafting deeper whenever acceptance is high is close to free upside: the 4.7× result above is adaptive doing exactly that. Qwen3.8-Flash-Next's own MTP head is a heavier draft, and getting adaptive drafting to pay off with it took further work in our fork's speculative decoding pipeline, not just the acceptance-tracking controller. With that in place, we're now seeing up to 40 tok/s generation on a Strix Halo with the MTP head, adaptive included; see the models page for the exact flags.

A Vulkan backend built for Strix Halo

Speculative decoding runs at small batch sizes (3–8), a regime upstream llama.cpp's Vulkan backend wasn't tuned for. We fixed a register-spillover bug in the IQ3_S shader that was causing a 5× slowdown at those batch sizes, and padded LDS strides to eliminate bank conflicts, for a 7–18% gain on top.

Prefill got the same attention: +13% on dense models at shallow context, and +89% on mixture-of-experts models at ubatch 2048 (1.9× mainline throughput).

ROCmFPx: a quantization format upstream can't load

The fork also integrates ROCmFPx, a 4-bit floating-point quantization format ported from AMD's ROCm FP spec, letting weights compress further than the quant types mainline llama.cpp supports today. See how much that buys us on Qwen3.8-Flash-Next, or browse the builds themselves on the models page.

Status

Everything above lives on our fork today; an upstream contribution to llama.cpp is in progress. The code, benchmarks, and full flag documentation are on GitHub.