Instructions to use thomasgardiner/Qwen3.8-27B-DFlash2-FP8 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use thomasgardiner/Qwen3.8-27B-DFlash2-FP8 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="thomasgardiner/Qwen3.8-27B-DFlash2-FP8")

# Load model directly
from transformers import AutoTokenizer, AutoModel

tokenizer = AutoTokenizer.from_pretrained("thomasgardiner/Qwen3.8-27B-DFlash2-FP8")
model = AutoModel.from_pretrained("thomasgardiner/Qwen3.8-27B-DFlash2-FP8", device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use thomasgardiner/Qwen3.8-27B-DFlash2-FP8 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "thomasgardiner/Qwen3.8-27B-DFlash2-FP8"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "thomasgardiner/Qwen3.8-27B-DFlash2-FP8",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/thomasgardiner/Qwen3.8-27B-DFlash2-FP8

SGLang

How to use thomasgardiner/Qwen3.8-27B-DFlash2-FP8 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "thomasgardiner/Qwen3.8-27B-DFlash2-FP8" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "thomasgardiner/Qwen3.8-27B-DFlash2-FP8",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "thomasgardiner/Qwen3.8-27B-DFlash2-FP8" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "thomasgardiner/Qwen3.8-27B-DFlash2-FP8",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use thomasgardiner/Qwen3.8-27B-DFlash2-FP8 with Docker Model Runner:
```
docker model run hf.co/thomasgardiner/Qwen3.8-27B-DFlash2-FP8
```

Qwen3.8-27B-DFlash2-FP8

+4 to +6% decode throughput on an RTX 5090 over the bf16 draft at the same acceptance length.

incoai/Qwen3.8-27B-DFlash2 with FP8 MLP and o_proj. Draft for Qwen3.8-27B. Runs in SGLang.

Checkpoint

Base: incoai/Qwen3.8-27B-DFlash2, 5 layers, block size 8, bf16, 3.85 GB
This checkpoint: 2.41 GB. 20 tensors in float8_e4m3fn: layers.*.mlp.gate_proj, up_proj, down_proj (15) and layers.*.self_attn.o_proj (5). Weights use a symmetric static per-tensor scale; activations a symmetric dynamic per-tensor scale. Format: compressed-tensors float-quantized. No calibration data.
Unchanged (bf16): q_proj, k_proj, v_proj, fc, candidate_selector.*, convolutions, norms. The three attention input projections stay bf16 so SGLang's fused DFlash KV materialization remains enabled.
Target: Qwen3.8-27B in any precision. Measured against RadixArk/Qwen3.8-27B-NVFP4.

Acceptance length

Greedy, thinking enabled, max_tokens 1024, one request at a time. Acceptance is the token-weighted mean of SGLang's per-batch accept len; prompts under about 130 generated tokens produce no sample and are excluded.

Workload	Prompts	bf16 draft	FP8 draft	Change
MT-Bench, first turn	80	3.80	3.83	+0.8%
HumanEval	164	4.64	4.58	−1.3%
GSM8K, test	100	5.07	5.02	−1.0%
MATH-500	100	5.14	5.08	−1.2%

Greedy: a draft token is kept only on a target argmax match. Target tokens are unchanged.

Throughput

One RTX 5090, SGLang lmsysorg/sglang@sha256:616a3e97… with the b12x NVFP4 backend patch, target RadixArk/Qwen3.8-27B-NVFP4, 8 draft tokens.

Concurrency 1

Workload	bf16 draft	FP8 draft	Change
MT-Bench, first turn	190.5 tok/s	201.5 tok/s	+5.8%
HumanEval	239.2 tok/s	248.8 tok/s	+4.0%
GSM8K	264.0 tok/s	277.0 tok/s	+4.9%
MATH-500	261.6 tok/s	272.4 tok/s	+4.1%

Decode step from the server log, same three greedy prompts: 19.10 ms (bf16) → 18.31 ms (FP8). Not measured: concurrency above 1, engines other than SGLang.

Serving with SGLang

python3 -m sglang.launch_server \
  --model-path RadixArk/Qwen3.8-27B-NVFP4 \
  --speculative-algorithm DFLASH \
  --speculative-draft-model-path thomasgardiner/Qwen3.8-27B-DFlash2-FP8 \
  --speculative-num-draft-tokens 8 \
  --attention-backend flashinfer \
  --trust-remote-code \
  --max-running-requests 1 \
  --cuda-graph-max-bs-decode 1 \
  --mem-fraction-static 0.91

SGLang reads the quantization config from the checkpoint. Do not pass --speculative-draft-model-quantization. The server log reports DFLASH fused KV materialization enabled when the fused path is active.

Creation

python3 quantize.py <incoai/Qwen3.8-27B-DFlash2 dir> <output dir> tensor

quantize.py in this repository.

Artifact identity

File	Size	sha256 (first 16)
model.safetensors	2.41 GB	1f3636a32d866f8e

Apache-2.0. DFlash 2 and the base draft are by inco.ai and z-lab.

Downloads last month: -

Safetensors

Model size

2B params

Tensor type

BF16

F8_E4M3

Model tree for thomasgardiner/Qwen3.8-27B-DFlash2-FP8

Base model

Qwen/Qwen3.8-27B

Quantized

(984)

this model

Collection including thomasgardiner/Qwen3.8-27B-DFlash2-FP8

Qwen3.8-27B on RTX 5090

Collection

SGLang b12x patch, all-NVFP4 target, FP8 DFlash2 draft. +52% decode. Scripts: github.com/thomgardiner/sglang-sm120-nvfp4 • 4 items • Updated about 2 hours ago