The Transformers modeling backend serves DeepSeek-R1 at native speed

vLLM PR #48250 (MLA fuser) + PR #48757 (fusion passes)

The MLA fuser maps Hugging Face's Multi-head Latent Attention onto vLLM's optimized MLA layer, so a DeepSeek model served through --model-impl transformers gets the same compressed-latent KV cache as the hand-written native implementation. Measured on DeepSeek-R1-0528 — 671B parameters, 37B active, native FP8 — across 16×H100-80GB under 3D parallelism (TP8 × PP2 × EP8), the Transformers backend runs at 99.6% of native throughput averaged over six workloads, with every point inside 3%.

Results (% of native vLLM)

Concurrency ladder, 1024-token prompts, 2048-token generations — R1 is a reasoning model, so the workload is a moderate prompt and a long chain of thought.

Mean 99.6% of native (range 98.0–102.8%). Decode latency tracks just as closely: mean TPOT is 100.4% of native across the five concurrency rungs. Time to first token is 98.8% of native on the same mean.

Each point is a single run — no repeats — so sub-percent differences are not signal. The unbounded leg is the largest deviation in either direction; the Transformers backend finishing 2.8% ahead there is not a claimed speedup, only evidence that the gap does not open up under saturation.

Both conditions run identical server flags, identical workloads, and the same transformers library from source; only --model-impl differs (native additionally needs --trust-remote-code, since it reads DeepSeek's config through its remote-code auto_map while the Transformers backend uses the in-library modeling).

What the fuser actually replaced

The engine logs what it matched at load. On the Transformers path:

Using FLASH_ATTN_MLA attention backend
  Fused: gate + experts + shared experts (DeepseekV3MoE) -> MoERunner (internal routing)
  Fused: input_layernorm (DeepseekV3RMSNorm) -> RMSNorm
  Fused: kv_a_layernorm (DeepseekV3RMSNorm) -> RMSNorm
  Fused: post_attention_layernorm (DeepseekV3RMSNorm) -> RMSNorm
  Fused: q_a_layernorm (DeepseekV3RMSNorm) -> RMSNorm
  Fused: self_attn (DeepseekV3Attention) -> MLAAttention
Enabled custom fusions: norm_quant, act_quant, allreduce_rms

Native selects the same FLASH_ATTN_MLA backend and the same three custom fusions. That is the point: after fusing, the two paths are running the same attention, GEMM and MoE kernels, and the remaining difference is bookkeeping.

Topology: 3D parallelism, TP8 × PP2 × EP8

MLA sets num_kv_heads = 1, so tensor parallelism cannot shard the KV cache — TP replicates the latent cache on every rank. The usual R1 recipe is therefore pure data parallel with expert parallelism (TP1 + DP16 + EP), which costs nothing on KV.

That does not fit in 80 GB. DP replicates R1's non-expert weights on every rank, putting weights at 68.4 GiB/GPU and leaving 10.8 GiB where the KV cache wanted 29.2 GiB. This is an H100 constraint, not a fundamental one — on H200s the same shape has headroom to spare.

What fits is all three axes at once: tensor parallel 8, pipeline parallel 2, expert parallel 8 — 8-way TP within each node, the layers split across the two nodes, and R1's 256 experts sharded across the 8 ranks of each pipeline stage. The cost is an 8× replicated MLA latent cache, which is what the 4.31× / 3.87× concurrency figures above already reflect.

Reproduce

Two nodes of 8×H100-80GB. Start the engine on each node — node 0 hosts the API server, node 1 runs headless:

# node 0
vllm serve deepseek-ai/DeepSeek-R1-0528 --model-impl transformers \
  --tensor-parallel-size 8 --pipeline-parallel-size 2 --enable-expert-parallel \
  --all2all-backend deepep_high_throughput \
  --nnodes 2 --node-rank 0 --master-addr $NODE0 --master-port 29501 \
  --no-enable-prefix-caching --gpu-memory-utilization 0.9 --port 8241

# node 1 — same flags, plus --headless
vllm serve deepseek-ai/DeepSeek-R1-0528 --model-impl transformers \
  --tensor-parallel-size 8 --pipeline-parallel-size 2 --enable-expert-parallel \
  --all2all-backend deepep_high_throughput \
  --nnodes 2 --node-rank 1 --master-addr $NODE0 --master-port 29501 \
  --no-enable-prefix-caching --gpu-memory-utilization 0.9 --headless

Then benchmark against it. This is the unbounded row; the other rungs are the same command with --max-concurrency set to 1, 16, 64 or 128, and the long-context leg is --random-input-len 8192 --random-output-len 1024 --max-concurrency 32 --num-prompts 96:

vllm bench serve --model deepseek-ai/DeepSeek-R1-0528 --port 8241 \
  --dataset-name random --random-input-len 1024 --random-output-len 2048 \
  --num-prompts 512 --ignore-eos \
  --percentile-metrics ttft,tpot,itl,e2el --save-result --result-filename tfm.json

For the native reference, swap --model-impl transformers for --model-impl vllm --trust-remote-code and restart the server — native reads DeepSeek's config through its remote-code auto_map, while the Transformers backend uses the in-library modeling. Nothing else changes.

Leave --max-model-len unset so the engine takes R1's own 163,840; capping it would inflate the reported concurrency multiple, which is the stat the MLA comparison quotes.

Raw results (vllm bench output)

{
  "date": "2026-07-28T15:03:06Z",
  "endpoint_type": "openai",
  "backend": "openai",
  "model_id": "deepseek-ai/DeepSeek-R1-0528",
  "num_prompts": 512,
  "request_rate": "inf",
  "burstiness": 1.0,
  "max_concurrency": null,
  "duration": 285.245001591,
  "completed": 512,
  "failed": 0,
  "total_input_tokens": 523683,
  "total_output_tokens": 1048576,
  "request_throughput": 1.794948192410866,
  "input_throughput": 1835.9059653248037,
  "output_throughput": 3676.0538980574534,
  "total_token_throughput": 5511.959863382257,
  "max_output_tokens_per_s": 7296.0,
  "max_concurrent_requests": 512,
  "mean_ttft_ms": 12650.083188916024,
  "median_ttft_ms": 12589.589538999999,
  "std_ttft_ms": 7196.290916923024,
  "p99_ttft_ms": 25139.36458525,
  "mean_tpot_ms": 98.77910820285302,
  "median_tpot_ms": 100.00085775769418,
  "std_tpot_ms": 21.35193265558169,
  "p99_tpot_ms": 127.0218162356668,
  "mean_itl_ms": 98.77910820285261,
  "median_itl_ms": 64.573471,
  "p99_itl_ms": 379.100455,
  "mean_e2el_ms": 214850.9176801562,
  "median_e2el_ms": 217291.41715999998,
  "p99_e2el_ms": 285038.57440309995
}

{
  "date": "2026-07-28T15:45:13Z",
  "endpoint_type": "openai",
  "backend": "openai",
  "model_id": "deepseek-ai/DeepSeek-R1-0528",
  "num_prompts": 512,
  "request_rate": "inf",
  "burstiness": 1.0,
  "max_concurrency": null,
  "duration": 277.429484294,
  "completed": 512,
  "failed": 0,
  "total_input_tokens": 523683,
  "total_output_tokens": 1048576,
  "request_throughput": 1.8455140098138194,
  "input_throughput": 1887.6256117213484,
  "output_throughput": 3779.612692098702,
  "total_token_throughput": 5667.2383038200505,
  "max_output_tokens_per_s": 7680.0,
  "max_concurrent_requests": 512,
  "mean_ttft_ms": 12596.19438248046,
  "median_ttft_ms": 12539.140319000002,
  "std_ttft_ms": 7158.228256194002,
  "p99_ttft_ms": 25022.91038557,
  "mean_tpot_ms": 96.35147293843127,
  "median_tpot_ms": 92.69548367024913,
  "std_tpot_ms": 18.672019963976954,
  "p99_tpot_ms": 123.25109050512458,
  "mean_itl_ms": 96.35147293843056,
  "median_itl_ms": 67.474756,
  "p99_itl_ms": 377.44121772999995,
  "mean_e2el_ms": 209827.6594874493,
  "median_e2el_ms": 202286.947066,
  "p99_e2el_ms": 277200.33659341006
}

Every number in the results table above comes from the vllm bench serve JSON for that run.