Qwen3.8-Flash-Next
Requants and hardware-specific builds of Qwen3.8-Flash-Next, published on HuggingFace. I think this is one of the strongest quantized lineups available for this model, at any size. My newest tier, AP-IQ2_S, is the smallest one I ship, and it outperforms the leading community build in its size class. See the research writeup for how I measured it, or the ROCmFPx writeup for how it buys back size elsewhere in the lineup.
My smallest build beats the leading community build in the same size class by 2.78 points of top-1 agreement, at roughly the same file size. Full methodology in the research writeup.
| Build | Format | Size | Top-1 agreement* | Target |
|---|---|---|---|---|
| AP-IQ2_S (smallest) | GGUF, mainline-compatible | 76.03 GiB | 79.14% | Any llama.cpp build, -ngl 99 |
| AP-IQ3_XXS | GGUF, mainline-compatible | 80.72 GiB | 81.85% | Any llama.cpp build, -ngl 99 |
| AP-IQ4_XS | GGUF, mainline-compatible | 84.24 GiB | 82.89% | Any llama.cpp build, -ngl 99 |
| ROCmFP4-FAST-imatrix | ROCmFPx, 4.23 bpw | 87.06 GiB | 84.56% | Strix Halo (Radeon 8060S, 96 GB unified) |
| AP-Q4_K_XL | GGUF, mainline-compatible | 94.20 GiB | 85.71% | Any llama.cpp build, -ngl 99 |
| AP-Q5_K_M | GGUF, mainline-compatible | 103.57 GiB | 86.48% | Any llama.cpp build, -ngl 99 |
| AP-Q5_K_XL (recommended) | GGUF, mainline-compatible | 112.51 GiB | 86.63% | Any llama.cpp build, -ngl 99 |
*Top-1 agreement is how often a build picks the same token as an unquantized Q8_0 reference, measured on a held-out corpus the model hasn't seen. I moved off wikitext-2 perplexity because this model's per-layer n-gram table has memorized it; see the writeup for why, including for my own earlier numbers. "AP" is Agention Precision, a per-layer compression profile, distinct from the ROCmFPx format used in the Strix Halo build.
Size for size, I haven't found a mainline-compatible lineup for this model that beats these numbers on the same benchmark. If you have, I'd genuinely like to see it, and I'll measure it the same way I measured everything above.
Size vs. quality across the lineup
The same relationship holds across every tier I publish: at each size where I have a matching unsloth tier, mine measures higher top-1 agreement. ROCmFP4-FAST, my fork-only build for Strix Halo, lands on the same curve as the mainline AP tiers rather than above it; its case for existing is speed, not extra quality-per-byte, see the research writeup for the full comparison.
Every point measured on the same Q8_0 reference and the same held-out corpus. Hover or tap a point for exact figures; full numbers are in the table above.
Running these
The AP builds are stock GGUF: any current llama.cpp build loads the weights. Speculative decoding needs my fork, since MTP support for this model isn't upstream yet; the draft head adapts its length to measured acceptance instead of guessing a fixed depth. If you're VRAM-constrained, AP-IQ2_S runs in roughly two-thirds the footprint of AP-Q5_K_XL and still leads the field in its size class; swap the model path below and the same flags apply.
llama-server -m AP-Q5_K_XL/Qwen3.8-Flash-Next-AP-Q5_K_XL.gguf \
-md Qwen3.8-Flash-Next-MTP-Q8_0.gguf \
-ngl 99 --n-gpu-layers-draft 99 \
--spec-type draft-mtp --spec-draft-adaptive \
--spec-draft-n-min 2 --spec-draft-n-max 4ROCmFP4-FAST needs my fork: it adds the ROCmFPx quant types and the per-head n-gram layout, plus the Vulkan fixes that keep throughput from collapsing at long context. On the same file, at 32k context, that's 2.75× the prefill and 1.9× the generation speed of stock llama.cpp; stock decays 55% in generation from 512 to 32k tokens, this fork only 10%. With the speculative MTP head on top, I've seen up to 40 tok/s generation on a Strix Halo.
git clone https://github.com/LaurentZuijdwijk/llama.cpp
cd llama.cpp && git checkout vulkan/qwen4exp-rocmfpx
cmake -B build -DGGML_VULKAN=ON -DCMAKE_BUILD_TYPE=Release
cmake --build build -j
./build/bin/llama-server \
-m Qwen3.8-Flash-Next-ROCmFP4-FAST-v2-ple16.gguf \
-ngl 99 -ctk q8_0 -ctv q8_0 -fa onFull setup, vision, and speculative-decoding instructions are in each repo's README on HuggingFace.