Gyro 3.8 Flash Next

• 01Fits a 32 GB card. Gyro-S needs 27.64 GiB of GPU memory for its weights and 28.8 GiB with 64k of context. The model's n-gram table stays on disk.
• 023-bit quality at 2-bit size. Gyro-M (34.98 GiB) matches ISTA-DASLab IQ3_XXS (43.80 GiB) on KL divergence, 0.306 vs 0.311.
• 03Finishes what it starts. In ten long-reasoning coding runs, Gyro-M never looped and finished all ten; a 36.5 GiB 2-bit build looped in seven and finished six.
• 04Fast on one card. Up to 221 tok/s decode for Gyro-S on an RTX 5090 in Strata (140 on prose, ~4,900 tok/s prefill); 113 tok/s on agentionai/llama.cpp (CUDA), up to 218 with the MTP draft.

Pick your file

Both files keep the model's architecture: all 48 layers, all 512 experts per layer. The routed experts are stored in a rotated basis with our own rotor code, Agention Precision Rotor (APR); Gyro-M gives the experts and the shared layers more bits. The files need our llama.cpp build or Strata (see Install); stock llama.cpp cannot load them.

The TQ1_0 / TQ2_0 in the file names is a size class for the Hub's file browser; the files use our own rotor formats, not the standard ternary types. Abliterated versions of both files are in a separate repo.

Quality

How closely each file follows the source, token by token: KL divergence and top-1 agreement against the unsloth Q8_0 source, on a held-out corpus of 2026 technical writing and code (-c 2048, 60 chunks, one binary for every file). GPU memory is the GPU-resident weights; the n-gram table stays on disk for every file.

We don't use wikitext for this model: its n-gram table has memorised it. See the research writeup for why.

Behaviour: long reasoning without loops

Low-bit quants of this model tend to loop in long reasoning: the same paragraph, again and again, until the context runs out. We test for it with one demanding one-shot coding prompt, a voxel pagoda scene in a single HTML file, at reasoning effort medium, over ten seeds (temperature 0.7, top-p 0.95, top-k 20, min-p 0, on an RTX A6000).

Speed by hardware and engine

Decode at batch size 1, in tokens per second. Where we list prose, JSON and code separately, the speed depends on how predictable the output is.

The MTP draft (cost-aware) pays most on predictable output: code, JSON and edits. On fast cards it can slow down prose and long reasoning, so try both.

Install

Gyro runs on our llama.cpp fork, agentionai/llama.cpp (Vulkan and CUDA), and on Strata. Tested on Linux with Vulkan (Strix Halo, R9700) and CUDA (RTX 5090, RTX A6000). ROCm, Windows and Apple (Metal) are untested or not supported yet.

1. The llama.cpp container

The quickest start. The container image ghcr.io/agentionai/agention-llama:server has our build ready; it runs through Vulkan on both AMD and NVIDIA.

hf download agentionai/Qwen3.8-Flash-Next-Gyro-GGUF Qwen3.8-Flash-Next-Gyro-S-TQ1_0.gguf \
  mtp-Qwen3.8-Flash-Next-draft.gguf mmproj-F16.gguf --local-dir ~/models

# AMD (and Intel) GPUs
docker run --rm -it --device /dev/dri --group-add "$(getent group render | cut -d: -f3)" \
  -v ~/models:/models -p 8080:8080 ghcr.io/agentionai/agention-llama:server \
  -m /models/Qwen3.8-Flash-Next-Gyro-S-TQ1_0.gguf -ngl 999 -fa on --jinja --ngram-on-disk \
  -c 65536 -ctk q8_0 -ctv q8_0 --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.05 \
  --mmproj /models/mmproj-F16.gguf

# NVIDIA GPUs: the same, with the NVIDIA container toolkit and its Vulkan driver
docker run --rm -it --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all \
  -v ~/models:/models -p 8080:8080 ghcr.io/agentionai/agention-llama:server \
  -m /models/Qwen3.8-Flash-Next-Gyro-S-TQ1_0.gguf -ngl 999 -fa on --jinja --ngram-on-disk \
  -c 65536 -ctk q8_0 -ctv q8_0 --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.05 \
  --mmproj /models/mmproj-F16.gguf

Then open http://localhost:8080. For Gyro-M, download and pass Qwen3.8-Flash-Next-Gyro-M-TQ2_0.gguf instead.

NVIDIA with Vulkan: many cloud GPU containers grant only compute,utility; then NVIDIA's Vulkan driver cannot start and llama.cpp silently falls back to the CPU (no usable GPU found). Keep -e NVIDIA_DRIVER_CAPABILITIES=all. Minimal images may also need libxext6 and libx11-6. For full speed on NVIDIA, build with CUDA (below).

2. Build from source

Needs Vulkan headers 1.4 or newer and glslc for the Vulkan build. On Ubuntu 22.04 the system packages are too old: install the LunarG Vulkan SDK and source its setup-env.sh first. NVIDIA: build with CUDA instead (CUDA toolkit 12.x+); the RTX 5090 and A6000 numbers above are from the CUDA build.

git clone https://github.com/agentionai/llama.cpp && cd llama.cpp
cmake -B build -DGGML_VULKAN=ON && cmake --build build -j      # NVIDIA: -DGGML_CUDA=ON instead (CUDA toolkit 12.x+)
./build/bin/llama-server -hf agentionai/Qwen3.8-Flash-Next-Gyro-GGUF:Gyro-S -ngl 999 -fa on --jinja --ngram-on-disk \
    -c 65536 -ctk q8_0 -ctv q8_0 --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.05

With -hf the vision projector downloads and loads automatically, and so does the MTP draft when you add --spec-type draft-mtp. Use :Gyro-M for the larger file.

3. Strata (release candidate)

Strata has Gyro support on the rc1 branch. It is the fastest way to run Gyro on an RTX 5090, and its expert cache runs Gyro on 24 GB cards. One script sets it up; the full guide is docs/GYRO.md.

git clone -b rc1 https://github.com/agentionai/Strata && cd Strata
tools/gyro_setup.sh --model S      # or --model M

Settings and tuning

• →Sampling. Qwen's thinking-mode settings plus min-p: --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.05. Min-p trims the low-probability tokens that a low-bit quant lifts slightly.
• →Keep --ngram-on-disk in every command, with the model on fast NVMe. It keeps the n-gram table off the GPU and reads its rows with a fast parallel reader. Plain memory-mapping (--lazy-mode on) roughly halves prompt-processing speed.
• →MTP draft for speculative decoding: download mtp-Qwen3.8-Flash-Next-draft.gguf from the repo (about 3.75 GiB of GPU memory) and add --spec-type draft-mtp -md mtp-Qwen3.8-Flash-Next-draft.gguf --spec-draft-n-max 6 --spec-draft-n-min 2 --spec-draft-mtp-vocab 32768. With -hf, leave out -md. On a single 32 GB card the draft fits next to Gyro-S up to 24k context with -ub 256 -ctkd q8_0 -ctvd q8_0.
• →Two GPUs? Put the whole model on one and the MTP draft on the other: --device Vulkan0 --device-draft Vulkan1. Splitting the model's layers across both cards makes them take turns and is slower than one card.
• →16–24 GB cards. --n-cpu-moe N keeps the experts of N layers in system RAM, about 0.55 GiB each. Start with N = 26 on 16 GB or N = 12 on 24 GB and lower it until the model just fits; drop --mmproj to save about 1 GiB. Speed depends on your RAM bandwidth and CPU. 32 GB of system RAM minimum, 64 GB recommended. Or use Strata's expert cache.
• →Ampere (RTX 30-series, A-series) with CUDA: the batched prompt path can crash (MUL_MAT_ID failed, illegal memory access) when several requests with longer prompts run at once. Until the fix ships, start the server with GGML_CUDA_TQ_MMQ=0 or --parallel 1.
• →Vision. Both files read images through mmproj-F16.gguf (about 1 GiB of GPU memory). To run text-only, drop --mmproj (with -m) or add --no-mmproj (with -hf).

Memory and context

What the GPU holds for Gyro-S (weights, KV cache, recurrent state and compute buffers), q8_0 KV cache, n-gram table on disk, one slot:

On a 32 GB card: up to 128k context without the MTP draft, 24k with it, or put the draft on a second GPU. Gyro-M needs about 37 GiB at 64k context.

Status and roadmap

• ✓Gyro-S and Gyro-M: available.
• ✓Abliterated Gyro-S and Gyro-M: available in their own repo.
• …Gyro-L: coming. About 50 GiB, for 64 GB machines.
• …Gyro Signal: coming. Shorter answers, no extra loops.
• …Strata support: release candidate.

Gyro is experimental. The format and kernels may still change, and the files will be re-published when they do. We'd like to hear how it runs on your card.

Links

• ↗Gyro on HuggingFace (Gyro-S, Gyro-M, MTP draft, vision projector)
• ↗Gyro abliterated on HuggingFace
• ↗agentionai/llama.cpp and the container ghcr.io/agentionai/agention-llama:server
• ↗Strata, rc1 and its Gyro guide
• →Our mainline-compatible Qwen3.8-Flash-Next builds (AP tiers, 76 GiB and up)
• ♥Sponsor AgentionAI on GitHub: a one-person team; GPU time funds the next quant.