The benchmark was memorized

I needed a benchmark I could trust

Every time I tune a quantization recipe, how many bits go on gate/up versus down, which layers get a bump, which quantizer to reach for on which tensor, I need a number that tells me whether the change actually helped. For Qwen3.8-Flash-Next, that number was wikitext-2 perplexity, the standard the whole quantization community uses. While validating a new evaluation baseline, one chunk of wikitext-2 came back at a perplexity of 1.36, against ~5.0 on every historical run of what looked like the same model.

It wasn't a bug. The model has memorized wikitext-2. Qwen3.8-Flash-Next carries a 51.2-billion-parameter per-layer n-gram table, 16 lookup heads with roughly 20 million entries each, and that's exactly the structure that would do it. Measured cleanly across a full run, the same build scores 3.68 perplexity on wikitext-2 and 6.47 on text I'm confident it has never seen, a 1.76x gap. Every recipe decision I'd made using wikitext-2 numbers had been made on a benchmark that wasn't really measuring quantization loss. It was measuring recall.

Building benchmark data of my own

I couldn't tune anything with confidence until I fixed that, so I assembled a held-out corpus from text with a hard cutoff after this model's training data: my own fork's engineering notes, llama.cpp commit messages since June 2026, and code diffs since August 2026, interleaved in 4KB blocks so no single source dominates any chunk count. And I stopped comparing quants to each other, which just measures agreement between two lossy builds, and instead compare every candidate against an unquantized Q8_0 reference, using KL-divergence and top-1 token agreement rather than perplexity alone. Perplexity is an average; it can hide a real change in which token wins when gains and losses cancel out. KL-divergence doesn't.

The corpus choice turned out not to be a house-brand advantage. I measured a competitor's build, unsloth's UD-IQ4_XS, on both corpora against the same Q8_0 reference: its wikitext-to-held-out discount is +7.96 points of top-1 agreement, almost identical to my own comparable-quality build's +6.85. The memorization is a property of the architecture, not of who quantized it, which told me the new benchmark was measuring the thing I actually cared about: how good is this recipe, not how well does it recite Wikipedia.

A case study: revisiting my smallest tier

To see what the new benchmark actually changed, I used it to revisit my IQ2-class recipe, the size class where every bit of budget matters most and a bad allocation shows up fastest. Below is how AP-IQ2_S compares to unsloth's UD-Q2_K_XL, the leading community build in the same size class, on identical tooling: same Q8_0 reference, same held-out corpus, same chunking.

unsloth UD-Q2_K_XL73.45 GiB

76.35 %

Agention AP-IQ2_S76.03 GiB

79.14 %

+2.78 points of top-1 agreement (11.7σ) and 21% lower KL-divergence (20.7σ). Correcting for the 2.58 GiB size difference along the local size/quality curve still leaves a gap of about 1.2 points at matched size, so the difference isn't just explained by AP-IQ2_S being the larger file.

The same benchmark also answered a question the old wikitext numbers couldn't settle cleanly: my Q5-class build, AP-Q5_K_M, reproduces unsloth's own published Q5 figures almost exactly (0.036 mean KLD, 93.2% top-1, versus their reported ~0.03 and ~93.5%). At that size the two recipes measure as equivalent. The gap I saw above is specific to the 2-bit tier, not a general pattern across sizes.

Widening out to every size I publish, the same relationship holds at each point where I have a matching unsloth tier. I also plotted ROCmFP4-FAST, my fork-only build for Strix Halo: it lands right on the mainline AP curve rather than above it, which is itself informative, it isn't buying extra quality-per-byte. Its n-gram table can be offloaded the same way the AP builds' can; this recipe just keeps it resident, at the cost of a slightly larger VRAM footprint. What it's actually trading that for is speed, a property of the ROCmFPx encoding itself, which runs substantially faster for its size than the mainline quant types do.

Every point measured on the same Q8_0 reference and the same held-out corpus. Hover or tap a point for exact figures; full numbers, including mean KL-divergence for every point, are in the table below.

What the constraint taught me

Digging into why the gap opens specifically at 2-bit, I traced it to a structural cause rather than a tuning one. One of the expert-projection tensors in this architecture has a shape that mainline llama.cpp's lowest-bit quantizer types can't target at all, so any quantizer, mine included, silently falls back to a higher-precision type on it regardless of what the rest of the recipe asks for. At a low overall bit budget that leaves the tensor comparatively over-provisioned next to its neighbors, and the neighbors correspondingly short-changed. I wouldn't have trusted a measurement this specific on the old benchmark; on the new one, it holds up at twenty sigma.

Once I could see that clearly, reallocating precision away from the over-provisioned tensor and toward the starved ones improved quality without changing the file size at all. I haven't published the exact per-tensor recipe that came out of this, it's still the part I'm actively iterating on, but the general lesson is one I didn't expect going in: how much a fixed recipe leaves on the table depends more on a tensor's shape and position in the architecture than on how large it is. Applying the same bit budget uniformly across a model is rarely the best use of it.

Where this leaves me

The held-out corpus is now how I evaluate every recipe change on this model, not just the one in this case study. AP-IQ2_S and the rest of the AP family are available as stock GGUF; the methodology and corpus behind the numbers in this piece are documented alongside the builds.

Browse the builds on HuggingFace ↗