Collaboration Call: High-Compute Scaling for SPECTRA-K3
Goal
We want independent high-compute contributors to determine which parts of the SPECTRA-MoE compression program remain viable when the dimensionality approaches real Kimi K3 expert tensors and when the work is distributed across modern accelerators.
The project is specifically seeking access to H100/H200, B100/B200, GB200/GB300, MI300X-class accelerators, 2/4/8/16+ GPU servers, and multi-node systems.
Current DGX Spark reference
The initial reference runs were completed on a single NVIDIA DGX Spark / GB10 with PyTorch 2.14.0+cu130 and CUDA 13.0.
Key measured operating points:
- 32768 x 32768, rank 64, 100 steps: 41.27B coefficients/s, 13.21 approximate TFLOP/s, 10.04 GiB peak accelerator memory.
- 8192 x 8192, rank 64, 16 experts, 100 steps: 15.30B coefficients/s, 4.90 approximate TFLOP/s, 10.05 GiB peak accelerator memory.
- At dimension 16384, residual energy improved from 0.2469 at rank 32 to 0.1555 at rank 256, demonstrating the expected sketch-quality/compute tradeoff.
These are synthetic compression-primitive measurements, not Kimi K3 task-quality results. The purpose of the collaboration is to extend these measurements to larger accelerators, multiple GPUs, multiple nodes, and eventually real K3 checkpoint analysis.
High-priority experiments
1. High-dimensional single-GPU sweeps
Run scripts/run_single_gpu_scaling.sh at the largest dimensions that fit cleanly. Particularly useful dimensions are 8192, 16384, 24576, and 32768 with sketch ranks 32/64/128/256.
2. Multi-GPU weak scaling
Keep EXPERTS per rank fixed and increase NPROC. This measures whether independent expert/shard analysis scales close to linearly when communication is limited to metric aggregation.
3. Multi-GPU strong scaling
Fix the global expert-block count and increase NPROC. This exposes launch, synchronization, and under-utilization limits.
4. K3 checkpoint inventory / shard audit
Contributors with sufficient storage can run the inventory against the released checkpoint and, when model-specific decoding is available, extend the spectral audit to real expert tensors.
5. Model-quality stages
The manuscript pre-registers a smaller open-MoE pilot before trillion-scale claims. Contributors interested in full model work are invited to add reproducible adapters for OLMoE, then K2/K2.5, and only then full K3.
What to submit
Please post the complete result JSON in a Hugging Face Discussion, together with:
- exact command used;
- accelerator model and count;
- whether the run was single-node or multi-node;
- driver/CUDA/ROCm/PyTorch versions;
- any changes to the benchmark code;
- OOMs, numerical failures, unsupported dtypes, or negative results.
Do not hand-edit performance numbers. The JSON is intended to be generated by the benchmark.
Contribution philosophy
This is a falsification-oriented benchmark. A result showing that a transform has no gain, that rank must be very high, that scaling saturates, or that a packed K3 tensor cannot be analyzed without a correct decoder is useful evidence.
Synthetic proxy throughput must not be described as K3 quality retention. Likewise, shard-level compressibility does not establish end-to-end task retention.
Suggested discussion title
[RESULT] <GPU x count> | <weak/strong/single> | dim=<D> rank=<R>
Example:
[RESULT] 8x H200 | weak | dim=16384 rank=128
Maintainer integration
The maintainer can copy accepted JSON files into results/raw/ and rebuild the table with:
python benchmarks/merge_results.py --input results/raw --output results/leaderboard.csv
No Git workflow is required for contributors to run the benchmark or post results.