The benchmark was memorized

We needed a benchmark we could trust

Every time we tune a quantization recipe, how many bits go on gate/up versus down, which layers get a bump, which quantizer to reach for on which tensor, we need a number that tells us whether the change actually helped. For Qwen3.8-Flash-Next, that number was wikitext-2 perplexity, the standard the whole quantization community uses. While validating a new evaluation baseline, one chunk of wikitext-2 came back at a perplexity of 1.36, against ~5.0 on every historical run of what looked like the same model.

It wasn't a bug. The model has memorized wikitext-2. Qwen3.8-Flash-Next carries a 51.2-billion-parameter per-layer n-gram table, 16 lookup heads with roughly 20 million entries each, and that's exactly the structure that would do it. Measured cleanly across a full run, the same build scores 3.68 perplexity on wikitext-2 and 6.47 on text we're confident it has never seen, a 1.76x gap. Every recipe decision we'd made using wikitext-2 numbers had been made on a benchmark that wasn't really measuring quantization loss. It was measuring recall.

Building benchmark data of our own

We couldn't tune anything with confidence until we fixed that, so we assembled a held-out corpus from text with a hard cutoff after this model's training data: our own fork's engineering notes, llama.cpp commit messages since June 2026, and code diffs since August 2026, interleaved in 4KB blocks so no single source dominates any chunk count. And we stopped comparing quants to each other, which just measures agreement between two lossy builds, and instead compare every candidate against an unquantized Q8_0 reference, using KL-divergence and top-1 token agreement rather than perplexity alone. Perplexity is an average; it can hide a real change in which token wins when gains and losses cancel out. KL-divergence doesn't.

The corpus choice turned out not to be a house-brand advantage. We measured a competitor's build, unsloth's UD-IQ4_XS, on both corpora against the same Q8_0 reference: its wikitext-to-held-out discount is +7.96 points of top-1 agreement, almost identical to our own comparable-quality build's +6.85. The memorization is a property of the architecture, not of who quantized it, which told us the new benchmark was measuring the thing we actually cared about: how good is this recipe, not how well does it recite Wikipedia.

A case study: revisiting our smallest tier

To see what the new benchmark actually changed, we used it to revisit our IQ2-class recipe, the size class where every bit of budget matters most and a bad allocation shows up fastest. Below is how AP-IQ2_S compares to unsloth's UD-Q2_K_XL, the leading community build in the same size class, on identical tooling: same Q8_0 reference, same held-out corpus, same chunking.

unsloth UD-Q2_K_XL73.45 GiB

76.35 %

Agention AP-IQ2_S76.03 GiB

79.14 %

+2.78 points of top-1 agreement (11.7σ) and 21% lower KL-divergence (20.7σ). Correcting for the 2.58 GiB size difference along the local size/quality curve still leaves a gap of about 1.2 points at matched size, so the difference isn't just explained by AP-IQ2_S being the larger file.

The same benchmark also answered a question the old wikitext numbers couldn't settle cleanly: our Q5-class build, AP-Q5_K_M, reproduces unsloth's own published Q5 figures almost exactly (0.036 mean KLD, 93.2% top-1, versus their reported ~0.03 and ~93.5%). At that size the two recipes measure as equivalent. The gap we saw above is specific to the 2-bit tier, not a general pattern across sizes.

Widening out to every size we publish, the same relationship holds at each point where we have a matching unsloth tier. We also plotted ROCmFP4-FAST, our fork-only build for Strix Halo: it lands right on the mainline AP curve rather than above it, which is itself informative, it isn't buying extra quality-per-byte. Its n-gram table can be offloaded the same way the AP builds' can; this recipe just keeps it resident, at the cost of a slightly larger VRAM footprint. What it's actually trading that for is speed, a property of the ROCmFPx encoding itself, which runs substantially faster for its size than the mainline quant types do.

Every point measured on the same Q8_0 reference and the same held-out corpus. Hover or tap a point for exact figures; full numbers, including mean KL-divergence for every point, are in the table below.

What the constraint taught us

Digging into why the gap opens specifically at 2-bit, we traced it to a structural cause rather than a tuning one. One of the expert-projection tensors in this architecture has a shape that mainline llama.cpp's lowest-bit quantizer types can't target at all, so any quantizer, ours included, silently falls back to a higher-precision type on it regardless of what the rest of the recipe asks for. At a low overall bit budget that leaves the tensor comparatively over-provisioned next to its neighbors, and the neighbors correspondingly short-changed. We wouldn't have trusted a measurement this specific on the old benchmark; on the new one, it holds up at twenty sigma.

Once we could see that clearly, reallocating precision away from the over-provisioned tensor and toward the starved ones improved quality without changing the file size at all. We haven't published the exact per-tensor recipe that came out of this, it's still the part we're actively iterating on, but the general lesson is one we didn't expect going in: how much a fixed recipe leaves on the table depends more on a tensor's shape and position in the architecture than on how large it is. Applying the same bit budget uniformly across a model is rarely the best use of it.

Where this leaves us

The held-out corpus is now how we evaluate every recipe change on this model, not just the one in this case study. AP-IQ2_S and the rest of the AP family are available as stock GGUF; the methodology and corpus behind the numbers in this piece are documented alongside the builds.

Browse the builds on HuggingFace ↗