Instructions to use unsloth/Qwen3-Coder-Next-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use unsloth/Qwen3-Coder-Next-GGUF with llama-cpp-python:

# !pip install llama-cpp-python

from llama_cpp import Llama

llm = Llama.from_pretrained(
	repo_id="unsloth/Qwen3-Coder-Next-GGUF",
	filename="BF16/Qwen3-Coder-Next-BF16-00001-of-00004.gguf",
)

llm.create_chat_completion(
	messages = [
		{
			"role": "user",
			"content": "What is the capital of France?"
		}
	]
)

Notebooks
Google Colab
Kaggle
Local Apps

llama.cpp

How to use unsloth/Qwen3-Coder-Next-GGUF with llama.cpp:

Install from brew

brew install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama-server -hf unsloth/Qwen3-Coder-Next-GGUF:UD-Q4_K_M
# Run inference directly in the terminal:
llama-cli -hf unsloth/Qwen3-Coder-Next-GGUF:UD-Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama-server -hf unsloth/Qwen3-Coder-Next-GGUF:UD-Q4_K_M
# Run inference directly in the terminal:
llama-cli -hf unsloth/Qwen3-Coder-Next-GGUF:UD-Q4_K_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf unsloth/Qwen3-Coder-Next-GGUF:UD-Q4_K_M
# Run inference directly in the terminal:
./llama-cli -hf unsloth/Qwen3-Coder-Next-GGUF:UD-Q4_K_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf unsloth/Qwen3-Coder-Next-GGUF:UD-Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf unsloth/Qwen3-Coder-Next-GGUF:UD-Q4_K_M

Use Docker

docker model run hf.co/unsloth/Qwen3-Coder-Next-GGUF:UD-Q4_K_M

LM Studio
Jan

vLLM

How to use unsloth/Qwen3-Coder-Next-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "unsloth/Qwen3-Coder-Next-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "unsloth/Qwen3-Coder-Next-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/unsloth/Qwen3-Coder-Next-GGUF:UD-Q4_K_M

Ollama
How to use unsloth/Qwen3-Coder-Next-GGUF with Ollama:
```
ollama run hf.co/unsloth/Qwen3-Coder-Next-GGUF:UD-Q4_K_M
```

Unsloth Studio new

How to use unsloth/Qwen3-Coder-Next-GGUF with Unsloth Studio:

Install Unsloth Studio (macOS, Linux, WSL)

curl -fsSL https://unsloth.ai/install.sh | sh
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for unsloth/Qwen3-Coder-Next-GGUF to start chatting

Install Unsloth Studio (Windows)

irm https://unsloth.ai/install.ps1 | iex
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for unsloth/Qwen3-Coder-Next-GGUF to start chatting

Using HuggingFace Spaces for Unsloth

# No setup required
# Open https://huggingface.co/spaces/unsloth/studio in your browser
# Search for unsloth/Qwen3-Coder-Next-GGUF to start chatting

Pi new

How to use unsloth/Qwen3-Coder-Next-GGUF with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama-server -hf unsloth/Qwen3-Coder-Next-GGUF:UD-Q4_K_M

Configure the model in Pi

# Install Pi:
npm install -g @mariozechner/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "unsloth/Qwen3-Coder-Next-GGUF:UD-Q4_K_M"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Hermes Agent new

How to use unsloth/Qwen3-Coder-Next-GGUF with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama-server -hf unsloth/Qwen3-Coder-Next-GGUF:UD-Q4_K_M

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default unsloth/Qwen3-Coder-Next-GGUF:UD-Q4_K_M

Run Hermes

hermes

Docker Model Runner
How to use unsloth/Qwen3-Coder-Next-GGUF with Docker Model Runner:
```
docker model run hf.co/unsloth/Qwen3-Coder-Next-GGUF:UD-Q4_K_M
```

Lemonade

How to use unsloth/Qwen3-Coder-Next-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull unsloth/Qwen3-Coder-Next-GGUF:UD-Q4_K_M

Run and chat with the model

lemonade run user.Qwen3-Coder-Next-GGUF-UD-Q4_K_M

List all available models

lemonade list

Huge thanks for the Qwen3-Coder-Next GGUFs 🙏

#30

by urbanswelt - opened 16 days ago

Discussion

urbanswelt

16 days ago

Just wanted to drop a quick appreciation post — your
Qwen3-Coder-Next-GGUF
release has turned my homelab into a genuine local-coding-agent platform.
Specifically running the UD-Q8_K_XL quant.

My setup

AMD Ryzen AI Max+ 395 ("Strix Halo") — integrated Radeon 8060S, 128 GiB
unified LPDDR5X memory
Fedora + ROCm 7.2.2 toolbox (kyuz0/amd-strix-halo-toolboxes)
llama.cpp in router mode via --models-preset (multi-model dropdown)
Quant: Qwen3-Coder-Next-UD-Q8_K_XL (~80 GiB, 3 shards)
Context: full 262144 native, no quality compromise

Performance — exceeds your published expectations

Your model card says: "If your quant fully fits on your device, expect 20+
tokens/s." On my Strix Halo box with the UD-Q8_K_XL quant, I'm seeing:

Test	Result
Single-stream tg128	36.72 t/s
Sustained tg at 64k context	36.6 t/s (no degradation)
Prompt processing (4k context)	668 t/s
4 parallel agents, 4k prompt + 256 gen each	76 t/s aggregate
8 parallel streams, small batches	102 t/s aggregate

So ~80% above your minimum expectation — and that's because the MoE
architecture (3B active out of 80B total) plays perfectly with this
platform's unified memory and integrated GPU. My measured memory bandwidth is
~217 GB/s (about 85% of theoretical), and a dense 80B model on this box would
crawl at maybe 2.5 t/s. Qwen3-Coder-Next runs 15× faster than a dense
model of equivalent quality would.

This is exactly the design promise from your release notes — "Super
Efficient with Significant Performance: With only 3B activated parameters
(80B total), it achieves performance comparable to models with 10–20× more
active parameters" — verified empirically on commodity hardware.

What's been excellent about your release specifically

Unsloth Dynamic 2.0 quants are noticeably better than vanilla quants at
the same file size. The UD- variants' selective bit allocation means I can
run Q8_K_XL and trust quality stays near-lossless. The third-party Aider /
LiveCodeBench / MMLU Pro benchmarks you link are genuinely useful — choosing
a quant from your repo doesn't feel like a coin flip.

The chat template fixes are not a small thing. With --jinja in
llama.cpp, tool-calling and the full 256k context "just work" — I tested 4
parallel coding-agent-style requests (4k context each, 256 token generation)
and they ran without any template wrangling. That's the kind of polish that
turns a model release into a usable product.

The model card and runbook are top-tier. Quant size guidance, hardware
requirements, sampling parameters, the docs.unsloth.ai run guide — I went
from "haven't tried this model" to running it in production-mode on novel
hardware in maybe 30 minutes. Compare that to the usual hours of figuring
out which quant to download and what flags to set.

Bottom line

Strix Halo gives me the memory pool. llama.cpp gives me the runtime. Your
quants give me the model that actually runs fast and produces good output.
All three pieces have to be good for the result to be good — and yours
delivers above the spec sheet.

For anyone else considering this combo: Qwen3-Coder-Next UD-Q8_K_XL on an
AMD Strix Halo (128 GiB) box is a serious local coding agent platform. 36
t/s single-stream, scales to 4 concurrent agents at 19 t/s each, full 256k
context, no thermal throttling under sustained load. With the right toolbox
the setup is straightforward.

Keep doing what you're doing. The Dynamic 2.0 work in particular is one of
those quiet contributions that makes the entire local LLM ecosystem better.

Thanks again 🙏

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment