khronnuz/Ling-3.0-flash-exl3

EXL3 quant of inclusionAI/Ling-3.0-flash at source revision ef06d91fe382109ae82647da88ff99b0f11745b0.

The source model card declares MIT. That repository does not ship a separate LICENSE file. This derivative keeps the declared license and does not add a license file the source did not include.

Stock ExLlamaV3 and TabbyAPI do not load these packs. They need the Ling-3.0-flash architecture patch proposed in ExLlamaV3 PR #419. The tested fork revision was a7b0515 (v1.5.1-61-ga7b0515, converter 1.5.1); the PR is not stock ExLlamaV3 support.

Quants

Pack	Decoder	HQ attention / shared experts	Output head
3.03 bpw / H8	3-bit	5-bit	8-bit
4.03 bpw / H8	4-bit	6-bit	8-bit

Requested decoder rates were 3.0 and 4.0 bpw with HQ; serialized allocation rates are 3.03 and 4.03 bpw. Both packs use the original weights, codebook mul1, output scales always, and a 250×2048 self-trace generated with an 8 bpw conversion. MTP is 8-bit, uncalibrated.

Download the selected quant branch, rather than main, and load it with the Ling-patched ExLlamaV3 fork described above.

Source sampling recommendation: temperature=0.6, top_p=0.95, top_k=20, thinking on by default.

Eight-document KL evaluation

Teacher-forced evaluation on eight documents: two each in English, Portuguese, Chinese, and Python. Each document contains 1,025 input token IDs, with 256 warmup positions and 768 scored next-token positions: 6,144 scored positions per candidate. Per-position KL is D_KL(p_original BF16 || q_EXL3) in nats, normalized across all 157,184 output-head columns, including unmapped tokenizer rows.

Candidate	Mean KL (nats)
3.03 bpw / H8	0.04015158961282057
4.03 bpw / H8	0.014791435550705148
8 bpw control	0.0029975456714980728

Independent acceptance verified the stored captures and recomputed all primary KL comparisons and wrong-row controls in NumPy FP64. Fresh within-backend replay was exact for the original and all three candidates.

These are short-window, eight-document engineering results. Original BF16 versus EXL3 FP16/mixed execution includes backend and precision differences, so KL is not pure weight-only quantization error. The 8 bpw control provides comparison context.