--- quantized_by: ubergarm pipeline_tag: text-generation base_model: stepfun-ai/Step-3.5-Flash base_model_relation: quantized license: apache-2.0 tags: - imatrix - conversational - ik_llama.cpp - step3p5 --- ## WIP Only one test quant for now, a custom `IQ4_XS` which runs on both mainline llama.cpp and [ik_llama.cpp now that this was just merged to main](https://github.com/ikawrakow/ik_llama.cpp/pull/1240). I'm cooking imatrix now and planning to release some more ik_llama.cpp quants on Saturday! ## `ik_llama.cpp` imatrix Quantizations of stepfun-ai/Step-3.5-Flash *NOTE* `ik_llama.cpp` can also run your existing GGUFs from bartowski, unsloth, mradermacher, etc if you want to try it out before downloading my quants. Some of ik's new quants are supported with [Nexesenex/croco.cpp](https://github.com/Nexesenex/croco.cpp) fork of KoboldCPP with Windows builds for CUDA 12.9. Also check for [Windows builds by Thireus here.](https://github.com/Thireus/ik_llama.cpp/releases) which have been CUDA 12.8. These quants provide best in class perplexity for the given memory footprint. ## Big Thanks Shout out to Wendell and the **Level1Techs** crew, the community [Forums](https://forum.level1techs.com/t/deepseek-deep-dive-r1-at-home/225826), [YouTube Channel](https://www.youtube.com/@Level1Techs)! **BIG thanks** for providing **BIG hardware** expertise and access to run these experiments and make these great quants available to the community!!! Also thanks to all the folks in the quanting and inferencing community on [BeaverAI Club Discord](https://huggingface.co/BeaverAI) and on [r/LocalLLaMA](https://www.reddit.com/r/LocalLLaMA/) for tips and tricks helping each other run, test, and benchmark all the fun new models! Thanks to huggingface for hosting all these big quants! Finally, I *really* appreciate the support from [aifoundry.org](https://aifoundry.org) so check out their open source RISC-V based solutions! ## Quant Collection Perplexity computed against *wiki.test.raw*. (lower is "better") ![Perplexity Chart](images/perplexity.png "Chart showing Perplexity vs Model Size.") These two are just a test quants for baseline perplexity comparison: * `BF16` 366.952 GiB (16.004 BPW) - TODO * `Q8_0` 195.031 GiB (8.506 BPW) - TODO *NOTE*: The first split file is much smaller on purpose to only contain metadata, its fine! ## IQ5_K TODO TODO
👈 Secret Recipe ```bash echo TODO ```
## IQ4_XS 100.53 GiB (4.38 BPW) TODO *NOTE*: This is the first test quant and does not use imatrix. It is compatible with mainline llama.cpp as well.
👈 Secret Recipe ```bash #!/usr/bin/env bash custom=" # 45 Repeating Layers [0-44] # Attention [0-44] GPU blk\..*\.attn_gate.*=q8_0 blk\..*\.attn_q.*=q8_0 blk\..*\.attn_k.*=q8_0 blk\..*\.attn_v.*=q8_0 blk\..*\.attn_output.*=q8_0 # First 3 Dense Layers [0-2] GPU blk\..*\.ffn_down\.weight=q8_0 blk\..*\.ffn_(gate|up)\.weight=q8_0 # Shared Expert Layers [3-44] GPU blk\..*\.ffn_down_shexp\.weight=q8_0 blk\..*\.ffn_(gate|up)_shexp\.weight=q8_0 # Routed Experts Layers [3-44] CPU blk\..*\.ffn_down_exps\.weight=iq4_xs blk\..*\.ffn_(gate|up)_exps\.weight=iq4_xs # Non-Repeating Layers token_embd\.weight=q4_K output\.weight=q6_K " custom=$( echo "$custom" | grep -v '^#' | \ sed -Ez 's:\n+:,:g;s:,$::;s:^,::' ) numactl -N ${SOCKET} -m ${SOCKET} \ ./build/bin/llama-quantize \ --custom-q "$custom" \ /mnt/data/models/ubergarm/Step-3.5-Flash-GGUF/Step-3.5-Flash-288x7.4B-BF16-00001-of-00009.gguf \ /mnt/data/models/ubergarm/Step-3.5-Flash-GGUF/Step-3.5-Flash-IQ4_XS.gguf \ IQ4_XS \ 128 ```
## IQ4_KSS TODO TODO
👈 Secret Recipe ```bash echo TODO ```
## IQ3_KS TODO TODO
👈 Secret Recipe ```bash echo TODO ```
## IQ2_KS TODO TODO
👈 Secret Recipe ```bash echo TODO ```
## Quick Start ```bash # Clone and checkout $ git clone https://github.com/ikawrakow/ik_llama.cpp $ cd ik_llama.cpp # Build for hybrid CPU+CUDA $ cmake -B build -DCMAKE_BUILD_TYPE=Release -DGGML_CUDA=ON $ cmake --build build --config Release -j $(nproc) ## https://github.com/ikawrakow/ik_llama.cpp/pull/1236 ## https://github.com/ikawrakow/ik_llama.cpp/pull/1231 ## https://github.com/ikawrakow/ik_llama.cpp/pull/1239 ## https://github.com/ikawrakow/ik_llama.cpp/pull/1240 echo TODO # CPU-only Mainline llama.cpp Example numactl -N "$SOCKET" -m "$SOCKET" \ ./build/bin/llama-server \ --model "$model"\ --alias ubergarm/Step-3.5-Flash \ --ctx-size 65536 \ -ctk q8_0 -ctv q8_0 \ -ub 4096 -b 4096 \ --parallel 1 \ --threads 96 \ --threads-batch 128 \ --numa numactl \ --host 127.0.0.1 \ --port 8080 \ --no-mmap \ --jinja ``` For tool use you can always bring your own template with `--jinja --chat-template-file myTemplate.jinja` and might need `--special` etc. The chat template baked into these GGUFs from the [original one](https://huggingface.co/stepfun-ai/Step-3.5-Flash/blob/main/chat_template.jinja). However just for tool use, it is possible [to copy paste the line out of this one](https://huggingface.co/stepfun-ai/Step-3.5-Flash-Int4/blob/main/step3p5_flash_Q4_K_S-00001-of-00012.gguf) but seems to mess it up for normal usage. Another option is to check out [pwilkin's autoparser branch](https://github.com/ggml-org/llama.cpp/pull/18675) which might work best in many cases. ## References * [ik_llama.cpp](https://github.com/ikawrakow/ik_llama.cpp) * [Getting Started Guide (already out of date lol)](https://github.com/ikawrakow/ik_llama.cpp/discussions/258) * [ubergarm-imatrix-calibration-corpus-v02.txt](https://gist.github.com/ubergarm/edfeb3ff9c6ec8b49e88cdf627b0711a?permalink_comment_id=5682584#gistcomment-5682584) * [mainline llama.cpp PR19283](https://github.com/ggml-org/llama.cpp/pull/19283) converted with `pull/19283/head:pr/step3.5-flash@5737bcf1b` plus casting `step35.attention.sliding_window_pattern` to `[INT32]` as for some reason it defaults to `[BOOL]` for me (which would work fine for mainline regardless). * [ik_llama.cpp PR1231](https://github.com/ikawrakow/ik_llama.cpp/pull/1231) imatrix & quantized with `ik/step35_compat@9a0b5e80`