Soomin33/Qwen3.8-Flash-Next-FP6

Qwen3.8-Flash-Next — FP6 (e2m3) experts + FP8 n-gram sidecar

Community quantization of Qwen3.8-Flash-Next targeting 2×64 GB GPUs with the 51B-parameter n-gram/PLE embedding table served from host RAM.

Provenance: this model was quantized from the official FP8 release (Qwen/Qwen3.8-Flash-Next-FP8, fine-grained fp8, block size 128) — not from the original BF16 weights. Tensors listed as bf16 below are bf16 containers holding values inherited from that FP8 checkpoint. The n-gram table ships in fp8 e4m3 in the official release; this repo only re-lays it out into a flat sidecar file — its values are unchanged.

Precision map

Verified from the safetensors headers of the released checkpoint.

Component	Params	Stored as	Size
MoE experts (gate/up/down, incl. the MTP layer experts)	123.31 B	fp6 e2m3, group 64, fp16 scales, hi4/lo2 uint8 planes (6.25 bits/weight)	89.7 GiB
GDN linear attention (36 layers)	2.09 B	bf16	3.9 GiB
Full attention + QSA indexer (12 layers)	0.81 B	bf16	1.5 GiB
Gated residual (hc) + misc	0.64 B	bf16	1.2 GiB
Token embeddings	0.64 B	bf16	1.2 GiB
lm_head (untied)	0.64 B	bf16	1.2 GiB
Vision encoder	0.31 B	bf16	0.6 GiB
Shared experts (not quantized)	0.24 B	bf16	0.45 GiB
MoE routers/gates	0.06 B	bf16	0.12 GiB
MTP draft, non-expert part	0.03 B	bf16	0.06 GiB
Sidecar: n-gram / PLE table (ple/ngram.bin)	51.2 B	raw fp8 e4m3, one global scale	47.7 GiB

Totals: 180 B parameters, ≈147 GiB on disk (≈6.6 bits/param checkpoint, ≈7.0 bits/param overall). No KV-cache or activation quantization anywhere; runtime uses bf16 KV and fp32 GDN/mamba state.

See FORMAT.md for the exact fp6 packing and sidecar layout.

Quantization quality

• The Triton fp6 dequant kernel is bit-exact against the reference encoder for all 64 codebook values.
• Expert-weight relative RMS error vs. the FP8 source: 0.0244 (predicted 0.0245 by the code-selection probe; e2m3 was chosen over e3m2 and int6 on measured GEMM-space error, not weight-space error).
• Fused MoE kernels validated against a torch reference on the same dequantized weights (max rel. err 3.2e-3 / 6.3e-3).

Benchmarks

GPQA Diamond, 45-question seeded subset (simple-evals prompt format, temperature 0, thinking enabled), served by this exact checkpoint on 2× CMP 170HX:

• Token budget set at 16,384. Of the questions it did answer, 100% were correct — 38/38. The remaining 7 exceeded the thinking budget and went unanswered rather than wrong.
• Official reference for the unquantized model: 91.7 (larger output budget).
• This is a quantization health check, NOT a leaderboard entry: n=45, single greedy run, one prompt format.

Full-run update (2026-08-28): a full GPQA Diamond pass at a 65,536-token output budget (temperature 0, truncation scored as incorrect) was stopped early by the operator at 66/71 = 93.0% (1 truncation) once it was clear the quantization is healthy — above the official 91.7 reference for the unquantized model, on a partial but seeded-order sample. The remaining questions and DeepSWE 1.1 stay on the backlog.

Hardware support

Built for and validated on NVIDIA Ampere (sm80, GA100 die) — the entire kernel set (fp6 dequant + fused MoE, PLE gather, QSA paths) is Triton with an sm80-safe fp6/fp8 decode written as integer bit manipulation, no fp8 tensor cores or Hopper/Blackwell intrinsics required. Anything an A100 can run, this runs; the validation hardware was the GA100-based CMP 170HX. Newer architectures (sm89/sm90/sm120) should work via the same Triton kernels but are untested here. Minimum footprint: 2× 64 GB GPUs (or ≥110 GB on one GPU) plus ≈50 GiB host RAM for the pinned n-gram sidecar.

Serving

Reference stack: SGLang (fork with the patches in patches/), 2× 64 GB sm80 GPUs (tested on CMP 170HX, PCIe gen2 x4), TP=2, plus ≈50 GiB of pinned host RAM for the sidecar (64 GiB system RAM minimum; the table is cudaHostAlloc pinned, not /dev/shm).

Serving architecture — what lives where

                 ┌───────────────────────────────────┐
                 │ Disk (≈147 GiB)                   │
                 │ 25 safetensors shards (≈99 GiB)   │
                 │ ple/ngram.bin       (≈48 GiB)     │
                 └─────────────────┬─────────────────┘
                                   │ read once at startup
          ┌────────────────────────┴────────────────────────┐
          ▼                                                 ▼
┌───────────────────────┐  2 all-reduces per layer  ┌───────────────────────┐
│ GPU 0 (64 GB, ≈58 in  │◄════════ PCIe ══════════► │ GPU 1 (64 GB, ≈58 in  │
│ use)                  │   (TP=2: each GPU holds   │ use)                  │
│  fp6 experts (half)   │    half of every tensor)  │  fp6 experts (half)   │
│  bf16 attn/GDN (half) │                           │  bf16 attn/GDN (half) │
│  KV cache, bf16       │                           │  KV cache, bf16       │
│  GDN state, fp32      │                           │  GDN state, fp32      │
└──────────┬────────────┘                           └──────────┬────────────┘
           │  zero-copy gather, ≈2.5 KB/token (16 rows × 160 B)│
           ▼                                                   ▼
┌─────────────────────────────────────────────────────────────────────────┐
│ Host RAM: n-gram / PLE table — fp8, 47.7 GiB, cudaHostAlloc-pinned      │
│ (vocab-sharded across the two ranks under TP; never occupies VRAM)      │
└─────────────────────────────────────────────────────────────────────────┘

Key properties of this layout:

• The 51B-parameter n-gram table never touches VRAM. It is a pure embedding lookup — never part of a matmul — so a GPU kernel gathers the ≈16 needed rows per token directly from pinned host memory over PCIe: ≈2.5 KB/token, microseconds even on a slow link. This is what lets a 180B-parameter model serve from 128 GB of VRAM.
• Compute weights (experts + attention) stay resident in VRAM at all times; KV cache is bf16 and GDN/mamba state fp32 — nothing about the cache or activations is quantized.
• Under PP (--pp-size 2 --tp-size 1) the split is by layers (0–23 / 24–47) instead of by tensor halves; the n-gram table then lives entirely with the stage that owns the PLE layer, and the inter-GPU traffic drops to one ≈20 KB hidden-state hop per token.
• Host RAM sizing: the pinned table is unreclaimable, so a 64 GiB box runs tight during weight loading — the reference launcher gates on MemAvailable and can wrap the server in a cgroup memory cap to keep weight-streaming page cache from stalling the machine.

Measured on the test box (CMP 170HX ×2, sm80, PCIe gen2 x4), with the kernel patches in patches/ (2026-08-29 update — see the next section for how these numbers roughly doubled):

• ≈27–28 tok/s single-stream plain decode (TP=2, no speculation), flat from short context to 47k (spot-checked deeper; long-context cost on this architecture is a constant, not a slope).
• ≈88 tok/s aggregate at 6 concurrent requests.
• ≈800–1,100 tok/s prefill (chunked, 2048).
• Pipeline parallelism (--pp-size 2 --tp-size 1) is supported by the included patches and verified output-identical; on this PCIe-only box it is slower than TP=2 (≈13 tok/s single) — PP pays off on NVLink-class links or >2 stages, not here.

How the speed doubled: 13.6–16.9 → 27–28 tok/s (instrumented)

The first release of this repo served at ≈17 tok/s short-context and 13.6 tok/s past 2k context. Phase-level instrumentation (timers included in the patches, env-gated) established two facts and led to three fixes — the full write-up with methodology and correctness gates is OPTIMIZATION-REPORT.md; the short version:

• The latency was never interconnect. A batch-1 token itemized as ≈1–2 ms of scheduler/transfer overhead, microseconds of n-gram sidecar gather from pinned host RAM, and ≈73 ms of GPU kernel chain. A control run of Qwen3.8-27B through the same machinery on the same PCIe gen2 x4 link pays <1 ms of overhead per token.
• Fix 1 — MoE tile size (BLOCK_M 64→32): the padded decode-row tile burned compute; bitwise-identical outputs, ≈24% faster MoE.
• Fix 2 — a GEMV-mode fp6 kernel for decode shapes (patches/ fp6_moe_kernels.py): no tl.dot padding waste, deduplicated packed-byte loads, one scale row per K-tile, fp32 accumulation (slightly more accurate than the tiled path). MoE block 1085 → ≈391 µs.
• Fix 3 — the "context decay" that wasn't: decode speed steps down at exactly ctx = 2048 (the QSA sparse-attention budget) and is flat beyond — and the step was ≈45 ms of per-call flash-attn CUTE-DSL dispatch overhead, not attention math. A dedicated Triton packed-decode attention kernel (patches/qsa_packed_decode.py, 31× faster at the same shape) removed it. Long context is now genuinely constant-cost: the same ≈27 tok/s at 47k as at 500.

Every change sits behind an env kill-switch and was gated by a 60-case kernel sweep against an fp32 reference, golden-drift bounds, determinism checks, greedy byte-identity, and a long-context needle probe (harnesses in tools/).

For buyers and optimizers: what remains between 28 and 50 tok/s is the GDN/indexer kernel chain — ordinary post-release kernel maturation, not bandwidth or interconnect. Better links move ≈1–2 ms of a ≈36 ms token; as fused kernels land upstream, this checkpoint gets faster on unchanged hardware.

Known limits of the reference stack (upstream issues, not weight issues):

• Speculative decoding (the built-in MTP head) is deliberately off in the reference config. On current SGLang trees the NEXTN path does run batched under TP=2 (measured 40–49 tok/s single-stream short-context, 32–43 at 20k), but it triples mamba-slot usage (effective concurrency ≈⅓) and its prefill cost has not been isolated (the one measurement was contaminated by concurrent traffic — treat it as unknown, not as a penalty). Greedy non-reproducibility exists with or without speculation (next bullet). The older EAGLE-as-own-draft path deadlocks batched under TP=2. PP and speculation are mutually exclusive in SGLang.
• Greedy (temp-0) decoding is not byte-reproducible run-to-run even without speculation: a tiny allocator-layout-dependent numeric noise in the decode path occasionally flips tokens where the model's top-2 choices are near-tied (~1% of steps; median top-2 margin is 5–8 nats). Measured effect on outcomes: none — 30/30 repeated greedy runs across a 6-question reasoning battery produced identical, correct final answers while the thinking paths varied. Both fp6 kernels in patches/ were verified bitwise-deterministic in isolation; the noise originates in stock decode paths and this class of behavior exists in most serving engines under concurrency anyway. Costs benchmark byte-reproducibility, nothing else observed.
• Speculative CUDA-graph capture has a flaky stream race on sm80.
• flash-attn-4 >= b28 required (b19 has a PackGQA compile bug on sm80).

Repository contents

• 25 safetensors shards + index, config.json, tokenizer files.
• ple/ngram.bin + ple/ngram-manifest.json — the sidecar and its layout.
• patches/ — SGLang fork patches (fp6 quant method, fused MoE kernels incl. the decode-shape GEMV variant, the Triton packed-decode attention kernel, PLE sidecar loader, qwen4_exp pipeline-parallel support, QSA indexer sync fix, env-gated phase timers).
• tools/ — fp6_encode.py (encoder), ple_extract.py (sidecar builder), fp6_probe.py (code-selection probe), test_fp6_sweep.py (60-case kernel correctness battery), soak_fp6.py (endless randomized soak), unit tests.
• launch-example.sh — reference SGLang launcher (parameterized version of the launcher this model was validated with).
• OPTIMIZATION-REPORT.md — full technical report on the 13.6→27 tok/s kernel work: measurements, fixes, negative results, correctness gates.
• QUANTIZING.md — reproducible procedure for converting any FP8 checkpoint of this family (fine-tunes included) into this format.
• LICENSE — Qwen Community License 1.0, inherited from the base model.
• FORMAT.md — byte-level format spec for third-party loader authors.