michael-qiu JustinTong commited on
Commit
6f040ee
·
1 Parent(s): 51de7e5

Trim SGLang launch command to the verified minimal recipe (#6)

Browse files

- Trim SGLang launch command to the verified minimal recipe (eaea50514aacc1e53fbf00c17326430e09c3bc43)


Co-authored-by: Xinyuan Tong <JustinTong@users.noreply.huggingface.co>

Files changed (1) hide show
  1. README.md +1 -2
README.md CHANGED
@@ -70,7 +70,6 @@ docker run --rm --gpus all --ipc=host --shm-size 32g \
70
  --tp 2 \
71
  --context-length 262144 \
72
  --json-model-override-args '{"rope_scaling":{"rope_type":"yarn","factor":2.0,"rope_theta":6000000,"partial_rotary_factor":0.5,"original_max_position_embeddings":131072}}' \
73
- --mem-fraction-static 0.85 \
74
  --trust-remote-code \
75
  --reasoning-parser auto \
76
  --tool-call-parser auto \
@@ -78,7 +77,7 @@ docker run --rm --gpus all --ipc=host --shm-size 32g \
78
  --port 30000
79
  ```
80
 
81
- On 80GB cards (H100 / H800), scale out to `--tp 8`. The reasoning and tool-call parsers resolve automatically to `ling3` from the chat template; you can also set them explicitly with `--reasoning-parser ling3 --tool-call-parser ling3`.
82
 
83
  **Client**
84
 
 
70
  --tp 2 \
71
  --context-length 262144 \
72
  --json-model-override-args '{"rope_scaling":{"rope_type":"yarn","factor":2.0,"rope_theta":6000000,"partial_rotary_factor":0.5,"original_max_position_embeddings":131072}}' \
 
73
  --trust-remote-code \
74
  --reasoning-parser auto \
75
  --tool-call-parser auto \
 
77
  --port 30000
78
  ```
79
 
80
+ On 80GB cards (H100 / H800), scale out to `--tp 8 --ep 8` (plain `--tp 8` fails the 128×128 block-quantization divisibility check; `--ep 8` keeps experts whole per rank). The reasoning and tool-call parsers resolve automatically to `ling3` from the chat template; you can also set them explicitly with `--reasoning-parser ling3 --tool-call-parser ling3`.
81
 
82
  **Client**
83