Working great on M5 Max 128GB — thanks!
Thank you for this quant — it runs beautifully out of the box with mlx-serve.
My setup: M5 Max, 128 GB unified memory, serving via mlx-serve with a 262K context window.
Real-world results:
Decode: ~70 tok/s at 7K context, ~54 at 28K, ~42 at 118K (official instruct sampling: temp 0.7 / top_p 0.8 / top_k 20); 90+ tok/s with greedy
Prefill: ~1100 tok/s at short context, still ~450 tok/s at 118K
Long agentic workloads are solid: 90K+ token prompts with multiple tool calls, steady 65–70 tok/s decode
MTP speculative decoding works great: ~3.6–4 accepted drafts per round, 90–100% draft hit rate
Tool calling has been reliable across many consecutive agent loops, no repetition or drift
Appreciate you sharing this — it's become my daily driver for local agent work.
Thank you for this quant — it runs beautifully out of the box with mlx-serve.
My setup: M5 Max, 128 GB unified memory, serving via mlx-serve with a 262K context window.
Real-world results:
Decode: ~70 tok/s at 7K context, ~54 at 28K, ~42 at 118K (official instruct sampling: temp 0.7 / top_p 0.8 / top_k 20); 90+ tok/s with greedy
Prefill: ~1100 tok/s at short context, still ~450 tok/s at 118K
Long agentic workloads are solid: 90K+ token prompts with multiple tool calls, steady 65–70 tok/s decode
MTP speculative decoding works great: ~3.6–4 accepted drafts per round, 90–100% draft hit rate
Tool calling has been reliable across many consecutive agent loops, no repetition or drift
Appreciate you sharing this — it's become my daily driver for local agent work.
I am building it now - other quants apparently don't support N-GRAM SSD streaming, or still very ram limited (OMLX quotes only 100K context can be supported)
Can you indicate what is a maximum context your mac (same as mine) can fit? I was hoping for 200k as tokens in this model should be smaller than massive 27b tokens where I can fit 250k due to much smaller checkpoint.
It totally survived loading full 262K context - awesome.
Real average PP speed (llama-benchy number is bogus for non-vllm or llamacpp engines) is around 262000/631 = 415 t/s which is fine for MLX, and 19 t/s at max context is great. It's not CUDA but lot better than anything I have seen yet on Metal.
PS: I am running 2x sparks stack, so used to fast pp and slower but stable generation across vast context (up to 1M real sessions).