ajh-code/Mage-VL-XPO3-NVFP4-W4A4

Mage-VL · XPO3 NVFP4 W4A4 · Stable v6.1

Stable still-image, paired-image, and 16-frame sampled-video understanding: native Blackwell W4A4 with direct packed decode, fused producer boundaries, and adaptive media-boundary scaling.

Our Quantization series combines the best of Speed, Quality, and Size.

Original Mage-VL · FP8 W8A8/W8A16 · Arands.com · updates

---

Download

Component	Purpose	Size
model-*.safetensors	Complete Mage-VL checkpoint	4.26 GB
streammind_gate.safetensors	Upstream proactive streaming gate	1.07 GB
Complete repository	Runtime, processor, codec support, and weights	5.62 GB

The root config.json is a recognized Hugging Face query file, so downloads are tracked and this repository appears as a quantization of microsoft/Mage-VL.

Stable v6.1 is a source/runtime update. The two model shards and StreamMind checkpoint are unchanged from the earlier release and do not need to be downloaded again when updating an existing local copy.

Versioning uses compact major/minor releases beginning with v6.1.

Quick start

Tested on Linux x86-64, Python 3.11, CUDA 13.1, PyTorch 2.13.0+cu130, comfy-kitchen==0.2.22, and NVIDIA Blackwell SM120.

hf download ajh-code/Mage-VL-XPO3-NVFP4-W4A4 --local-dir mage-vl-quant
cd mage-vl-quant
python3.11 -m venv .venv
. .venv/bin/activate
pip install -r requirements.txt

CUDA_HOME=/usr/local/cuda-13.1 CUDA_VISIBLE_DEVICES=0 \
  python inference.py --mode offline --model . --image image.jpg \
  --question "Describe this image in detail."

Stable v6.1 ships task-specific quality profiles plus every earlier fallback, without changing the checkpoint weights:

Profile	Decode policy	Intended use
pure_w4a4_v6_1_still (default)	Reference-order fused Q/K plus register-fed gate/up; ordinary tensor-wide prefill scaling	General still-image understanding; 40/40 greedy and 15/15 sampled outputs exact to the accepted V5 path
pure_w4a4_v6_1_edit	Stable V5 edit policy with 1,024-row chunk-local scaling in layers 12–17	Paired-image edit judgment; all 28 packaged outputs token/text exact to the accepted V5 edit path
pure_w4a4_v6_1_video	Adaptive short-video media-boundary scaling; exact V5 chunk-local fallback for longer answers	Stable 16-frame sampled video QA: 34/35 practically usable and 32/35 strict-complete
pure_w4a4_v5_still/edit/video	Frozen Runtime V5 task profiles	Exact rollback and comparison paths
pure_w4a4_v4_image	Complete V4 W4A4 stack, including tree-order fused Q/K norm-RoPE	Earlier fastest image profile
pure_w4a4_v4_video_safe	V4 with the original Q/K path	Conservative video experiments; 5/5 token-exact to V3 in the frozen gate
hybrid_fast	W4A4 prefill, packed-weight W4A16 at M=1	Earlier Runtime V2 hybrid fallback
pure_w4a4_v2	Shared-activation W4A4	Earlier Runtime V2 pure fallback

The bundled CLI automatically selects pure_w4a4_v6_1_video for video and samples 16 frames by default. Direct Transformers users should select the profile before loading. The profile applies the validated no_repeat_ngram_size=4 and repetition_penalty=1.05 guard unless overridden; pass --no-repeat-ngram-size 0 --repetition-penalty 1.0 to disable it:

MAGE_VL_RUNTIME_PROFILE=pure_w4a4_v6_1_video CUDA_VISIBLE_DEVICES=0 \
  python inference.py --mode offline --model . --video video.mp4 \
  --num-frames 16 --max-new-tokens 64 \
  --question "Which tool is used?"

Every V4/V5 boundary can be disabled independently without rebuilding weights:

MAGE_VL_V4_DIRECT_W4A4_M1
MAGE_VL_V4_INPUT_RMSNORM_PACK
MAGE_VL_V4_POST_RMSNORM_PACK
MAGE_VL_V4_EXACT_MLP_PRODUCT_PACK
MAGE_VL_V4_FUSED_QK_NORM_ROPE
MAGE_VL_V5_DIRECT_BACKEND=baseline|register_mlp
MAGE_VL_V5_QK_REDUCTION_ORDER=tree|reference
MAGE_VL_V5_CHUNKED_PREFILL=0|1
MAGE_VL_V5_PREFILL_CHUNK_ROWS
MAGE_VL_V5_PREFILL_CHUNK_LAYERS
MAGE_VL_V5_PREFILL_CHUNK_ROLES
MAGE_VL_V6_SEGMENTED_PREFILL=0|1
MAGE_VL_V6_MAX_VISUAL_GROUP_ROWS
MAGE_VL_V6_MIN_VISUAL_RUNS
MAGE_VL_V6_ROW_BUCKET_MULTIPLE
MAGE_VL_V6_PREFILL_LAYERS
MAGE_VL_V6_PREFILL_ROLES
MAGE_VL_V6_SHORT_FORM_MAX_NEW_TOKENS
MAGE_VL_V6_TOGGLE_BASE_CHUNK_BY_POLICY=0|1

Each accepts 1/0, true/false, yes/no, or on/off. Larger-M calls retain the established ordinary W4A4 path. The two V2 profiles remain packaged as known fallbacks and make regressions easy to isolate.

Measured performance

Stable v6.1 quality-recovery cost — RTX 5080

Repeated matched comparisons use the same checkpoint, 16 sampled frames, prompts, generation guard, and output budgets. An unrelated idle TTS service remained resident on the GPU, so these are conservative co-load measurements, not an unloaded maximum-performance claim.

Workload	Runtime V5	Stable v6.1	Result
Short video, 12-case median total	5.9214 s	5.9825 s	+1.03%
Short video, nine equal-token steady cases	3.6410 s	3.6654 s	+0.67%
Long video, three-case median total	9.3594 s	8.9979 s	no resolved slowdown
Paired edit, eight-case median total	2.4054 s	2.3893 s	no resolved slowdown
Still image, fixed 32 tokens	0.58570 s	unchanged V5 path	no added path

Do not interpret the apparent long-video/edit wins as claimed speedups; they are within fresh-process variability. The release claim is V5-class speed with roughly a one-percent cost only on the corrected short-video path.

Peak allocated memory is unchanged: 5,290.75 MiB for short video, 5,708.17 MiB for long video, and 4,546.60 MiB for paired edit.

Runtime V4 speed foundation — RTX 5080

Greedy decode, exactly 32 new tokens, SDPA, DynamicCache, compilation off, two warmups, 15 measured repeats. These are complete generate() medians, not isolated-kernel claims:

Frozen case	Runtime	Median	Resident allocation	Peak allocation
1024x576 lighthouse/OCR	BF16	0.69712 s	9,047.71 MiB	9,264.54 MiB
1024x576 lighthouse/OCR	Runtime V4 W4A4	0.58241 s	4,066.90 MiB	4,315.72 MiB
1280 cyborg/detail	Earlier W4A16 hybrid	0.71781 s	—	—
1280 cyborg/detail	Runtime V4 W4A4	0.67984 s	—	—

On the matched BF16 case, V4 is 1.197x as fast (16.45% lower latency) and uses 55.05% less resident model allocation. On the 1280 case it is 1.056x as fast as the earlier W4A16 hybrid (5.29% lower latency), making V4 the fastest Mage-VL profile we measured locally. This is not a claim that it is the fastest possible implementation across all runtimes, GPUs, batch sizes, prompts, or output lengths.

Cold start is different: the first post-load matched request measured 1.68008s for V4 versus 1.44271s for BF16 and the first V4 use JIT-compiles four small CUDA extensions. V4 targets a resident, repeatedly used service.

The full held-out quality workload took 60.08s under V4 versus 94.30s under V3, but those totals are not used as a benchmark because the generated output lengths differ.

Earlier three-way control — RTX 5060 Ti

The prior 1,657-token, fixed-32 protocol provides the BF16/FP8 context:

Runtime	Resident allocation	Prefill	Generate 32	Derived decode
BF16	9,044 MiB	0.6011 s	1.4277 s	36.69 tok/s
Ordinary FP8 W8A8	5,579 MiB	0.4541 s	2.0543 s	19.14 tok/s
FP8 W8A8/W8A16 fused	5,579 MiB	0.4535 s	1.0177 s	53.21 tok/s
XPO3 NVFP4 W4A4/W4A16	4,063 MiB	0.4233 s	1.0120 s	51.14 tok/s

These are matched local measurements, not universal end-to-end claims. On the 5060 Ti control, XPO3 was 1.41x faster than BF16 for fixed work and used 55.1% less resident model allocation.

How Runtime V4 got there

Runtime V4 is an optimization stack, not a new set of quantized weights:

• Direct packed M=1 W4A4: decode consumes packed activation and packed checkpoint data without falling back through a dequantized weight copy.
• RMSNorm-to-NVFP4 producer fusion: input and post-attention RMSNorm write the exact dynamic NVFP4 activation expected by the following projection.
• Exact MLP product packing: the BF16 SiLU(gate) * up boundary is packed directly for down_proj; it remained token-exact to V3 on all five frozen video cases.
• Fused Q/K head RMSNorm plus RoPE: one decode CUDA launch replaces the separate Q/K normalization and rotary steps. The primitive fell from 0.16521ms to 0.00712ms (23.21x), and the complete matched 1280 runtime fell from 0.89568s to 0.67984s (1.318x).

The Q/K fusion is the largest speed win, but it also caused the observed video answer drift. It is therefore on in the image profile and off in pure_w4a4_v4_video_safe. The exact MLP pack alone reproduced V3 tokens on all five frozen videos.

Rejected or non-beneficial paths included FlashAttention 2 for this workload, eager attention, static cache, torch.compile, CUDA Graph composition, grouped QKV/gate-up variants, replayed activation scales, and conventional narrow-M CUTLASS Tensor Core kernels. They remain out of the default path.

Both model shards, the checkpoint index, and StreamMind weights remain byte-for-byte unchanged. Existing users need only the updated Python, JSON, Markdown, and native/ source files.

Runtime V5 keeps that speed foundation and adds three bounded corrections: PyTorch-matching reference-order Q/K reduction, register-fed SM120 W4A4 for gate/up decode, and optional chunk-local activation scaling for edit/video prefills. Enabling chunking globally was rejected because it degraded long-form still descriptions; it is deliberately off in the default profile.

Stable v6.1 adds a fourth bounded correction. For video requests with max_new_tokens <= 64, it groups text and complete frame-token runs under separate dynamic NVFP4 activation scales in language layers 15–17. Longer answers retain the exact V5 video path. Broad/global boundary scaling was rejected because it introduced new long-answer errors. The runtime therefore activates this feature per request and restores every toggle afterward.

Quantization policy

All 252 Qwen3 language projections use the native packed NVFP4 checkpoint. Large-token prefill runs W4A4. In hybrid_fast, M=1 reads those same packed weights through a specialized BF16-activation/FP32-accumulation GEMV, avoiding activation-quantization and padded-GEMM overhead. In the V4 profiles, every quantized language-projection GEMM uses NVFP4 weights and NVFP4 activations. Exact dynamic activation sharing remains in place, while the V4/V5 producer fusions remove redundant conversions and kernel launches at M=1. No second weight copy is stored.

“Pure W4A4” describes those 252 language-projection GEMMs, not the complete end-to-end model: Mage-ViT, embeddings, LM head, normalization, RoPE, StreamMind, outputs, and accumulators remain at their documented higher precision. The Stable v6.1 still default retains the accepted PyTorch reduction order and is token-exact to V5 across the complete frozen still suite. This is not a claim of universal token identity to BF16; the remaining broader W4A4 wording differences are documented and semantically reviewed.

Validated scope

Gate	Result
Quantized language projections	252 / 252
Held-out still images	Stable v6.1 default is 40/40 greedy token tensors exact to accepted V5; corrected practical objective review remains 36/40
Sampled long-form images	Stable v6.1 default is 15/15 token tensors exact to accepted V5; 14/15 factual anchors under the frozen scorer
Paired-image edit reliability	All 28 packaged outputs token/text exact to accepted V5; practical decisions remain equal to BF16/FP8
Stable v6.1 video profile	34/35 practically usable and 32/35 strict-complete across 35 questions / 20 sequences; zero token caps and zero repeated four-gram failures
Sealed video final	9/12 for V5 and v6.1 with 12/12 answer-choice agreement
Packaging	Complete sharded Safetensors checkpoint with root config.json
Unchanged BF16 components	Mage-ViT, embeddings, LM head, norms, RoPE, StreamMind
Current hardware target	NVIDIA Blackwell SM120

Image, paired-image edit understanding, and 16-frame sampled video QA are validated release paths. The comprehensive video gate covered 35 questions across 20 sequences: 34/35 answers were practically usable and 32/35 fully satisfied the strict prompt. Stable v6.1 corrects the prior hand-tool and tutorial-project-name misses while preserving the other 33 accepted V5 outputs token exactly. Fine material identification remains the one practical miss. The decoder guard is part of the validated profile: disabling it reintroduced two token caps and 11 repetition or numeric-corruption failures.

In the earlier V5 sampling study, increasing to 32 frames was not monotonically better (30/35 practical versus 32/35 at 16 frames). Stable v6.1 is therefore validated at 16 frames; an adaptive 32-frame profile is not claimed. Audio understanding, streaming, and dense long-video analysis remain unvalidated. Use concise temporal questions and review answers when exact object names or fine materials are critical. Codec and StreamMind paths are retained for completeness.

Validate the download

python validate_release.py

MANIFEST.json records the size and SHA-256 of every distributed file except itself. Hashing the model and StreamMind checkpoints takes a little while.

License and attribution

Mage-VL and this derivative package are released under Apache-2.0. The model architecture, processor, codec utilities, and original BF16 weights derive from microsoft/Mage-VL. The quantized runtime modifications are identified in the bundled source and THIRD_PARTY_NOTICES.md.