---
quantized_by: ubergarm
pipeline_tag: text-generation
base_model: stepfun-ai/Step-3.5-Flash
base_model_relation: quantized
license: apache-2.0
tags:
- imatrix
- conversational
- ik_llama.cpp
- step3p5
---
## WIP
Only one test quant for now, a custom `IQ4_XS` which runs on both mainline llama.cpp and [ik_llama.cpp now that this was just merged to main](https://github.com/ikawrakow/ik_llama.cpp/pull/1240).
I'm cooking imatrix now and planning to release some more ik_llama.cpp quants on Saturday!
## `ik_llama.cpp` imatrix Quantizations of stepfun-ai/Step-3.5-Flash
*NOTE* `ik_llama.cpp` can also run your existing GGUFs from bartowski, unsloth, mradermacher, etc if you want to try it out before downloading my quants.
Some of ik's new quants are supported with [Nexesenex/croco.cpp](https://github.com/Nexesenex/croco.cpp) fork of KoboldCPP with Windows builds for CUDA 12.9. Also check for [Windows builds by Thireus here.](https://github.com/Thireus/ik_llama.cpp/releases) which have been CUDA 12.8.
These quants provide best in class perplexity for the given memory footprint.
## Big Thanks
Shout out to Wendell and the **Level1Techs** crew, the community [Forums](https://forum.level1techs.com/t/deepseek-deep-dive-r1-at-home/225826), [YouTube Channel](https://www.youtube.com/@Level1Techs)! **BIG thanks** for providing **BIG hardware** expertise and access to run these experiments and make these great quants available to the community!!!
Also thanks to all the folks in the quanting and inferencing community on [BeaverAI Club Discord](https://huggingface.co/BeaverAI) and on [r/LocalLLaMA](https://www.reddit.com/r/LocalLLaMA/) for tips and tricks helping each other run, test, and benchmark all the fun new models! Thanks to huggingface for hosting all these big quants!
Finally, I *really* appreciate the support from [aifoundry.org](https://aifoundry.org) so check out their open source RISC-V based solutions!
## Quant Collection
Perplexity computed against *wiki.test.raw*. (lower is "better")

These two are just a test quants for baseline perplexity comparison:
* `BF16` 366.952 GiB (16.004 BPW)
- TODO
* `Q8_0` 195.031 GiB (8.506 BPW)
- TODO
*NOTE*: The first split file is much smaller on purpose to only contain metadata, its fine!
## IQ5_K TODO
TODO
👈 Secret Recipe
```bash
echo TODO
```
## IQ4_XS 100.53 GiB (4.38 BPW)
TODO
*NOTE*: This is the first test quant and does not use imatrix. It is compatible with mainline llama.cpp as well.
👈 Secret Recipe
```bash
#!/usr/bin/env bash
custom="
# 45 Repeating Layers [0-44]
# Attention [0-44] GPU
blk\..*\.attn_gate.*=q8_0
blk\..*\.attn_q.*=q8_0
blk\..*\.attn_k.*=q8_0
blk\..*\.attn_v.*=q8_0
blk\..*\.attn_output.*=q8_0
# First 3 Dense Layers [0-2] GPU
blk\..*\.ffn_down\.weight=q8_0
blk\..*\.ffn_(gate|up)\.weight=q8_0
# Shared Expert Layers [3-44] GPU
blk\..*\.ffn_down_shexp\.weight=q8_0
blk\..*\.ffn_(gate|up)_shexp\.weight=q8_0
# Routed Experts Layers [3-44] CPU
blk\..*\.ffn_down_exps\.weight=iq4_xs
blk\..*\.ffn_(gate|up)_exps\.weight=iq4_xs
# Non-Repeating Layers
token_embd\.weight=q4_K
output\.weight=q6_K
"
custom=$(
echo "$custom" | grep -v '^#' | \
sed -Ez 's:\n+:,:g;s:,$::;s:^,::'
)
numactl -N ${SOCKET} -m ${SOCKET} \
./build/bin/llama-quantize \
--custom-q "$custom" \
/mnt/data/models/ubergarm/Step-3.5-Flash-GGUF/Step-3.5-Flash-288x7.4B-BF16-00001-of-00009.gguf \
/mnt/data/models/ubergarm/Step-3.5-Flash-GGUF/Step-3.5-Flash-IQ4_XS.gguf \
IQ4_XS \
128
```
## IQ4_KSS TODO
TODO
👈 Secret Recipe
```bash
echo TODO
```
## IQ3_KS TODO
TODO
👈 Secret Recipe
```bash
echo TODO
```
## IQ2_KS TODO
TODO
👈 Secret Recipe
```bash
echo TODO
```
## Quick Start
```bash
# Clone and checkout
$ git clone https://github.com/ikawrakow/ik_llama.cpp
$ cd ik_llama.cpp
# Build for hybrid CPU+CUDA
$ cmake -B build -DCMAKE_BUILD_TYPE=Release -DGGML_CUDA=ON
$ cmake --build build --config Release -j $(nproc)
## https://github.com/ikawrakow/ik_llama.cpp/pull/1236
## https://github.com/ikawrakow/ik_llama.cpp/pull/1231
## https://github.com/ikawrakow/ik_llama.cpp/pull/1239
## https://github.com/ikawrakow/ik_llama.cpp/pull/1240
echo TODO
# CPU-only Mainline llama.cpp Example
numactl -N "$SOCKET" -m "$SOCKET" \
./build/bin/llama-server \
--model "$model"\
--alias ubergarm/Step-3.5-Flash \
--ctx-size 65536 \
-ctk q8_0 -ctv q8_0 \
-ub 4096 -b 4096 \
--parallel 1 \
--threads 96 \
--threads-batch 128 \
--numa numactl \
--host 127.0.0.1 \
--port 8080 \
--no-mmap \
--jinja
```
For tool use you can always bring your own template with `--jinja --chat-template-file myTemplate.jinja` and might need `--special` etc. The chat template baked into these GGUFs from the [original one](https://huggingface.co/stepfun-ai/Step-3.5-Flash/blob/main/chat_template.jinja). However just for tool use, it is possible [to copy paste the line out of this one](https://huggingface.co/stepfun-ai/Step-3.5-Flash-Int4/blob/main/step3p5_flash_Q4_K_S-00001-of-00012.gguf) but seems to mess it up for normal usage.
Another option is to check out [pwilkin's autoparser branch](https://github.com/ggml-org/llama.cpp/pull/18675) which might work best in many cases.
## References
* [ik_llama.cpp](https://github.com/ikawrakow/ik_llama.cpp)
* [Getting Started Guide (already out of date lol)](https://github.com/ikawrakow/ik_llama.cpp/discussions/258)
* [ubergarm-imatrix-calibration-corpus-v02.txt](https://gist.github.com/ubergarm/edfeb3ff9c6ec8b49e88cdf627b0711a?permalink_comment_id=5682584#gistcomment-5682584)
* [mainline llama.cpp PR19283](https://github.com/ggml-org/llama.cpp/pull/19283) converted with `pull/19283/head:pr/step3.5-flash@5737bcf1b` plus casting `step35.attention.sliding_window_pattern` to `[INT32]` as for some reason it defaults to `[BOOL]` for me (which would work fine for mainline regardless).
* [ik_llama.cpp PR1231](https://github.com/ikawrakow/ik_llama.cpp/pull/1231) imatrix & quantized with `ik/step35_compat@9a0b5e80`