Commit ·
6f040ee
1
Parent(s): 51de7e5
Trim SGLang launch command to the verified minimal recipe (#6)
Browse files- Trim SGLang launch command to the verified minimal recipe (eaea50514aacc1e53fbf00c17326430e09c3bc43)
Co-authored-by: Xinyuan Tong <JustinTong@users.noreply.huggingface.co>
README.md
CHANGED
|
@@ -70,7 +70,6 @@ docker run --rm --gpus all --ipc=host --shm-size 32g \
|
|
| 70 |
--tp 2 \
|
| 71 |
--context-length 262144 \
|
| 72 |
--json-model-override-args '{"rope_scaling":{"rope_type":"yarn","factor":2.0,"rope_theta":6000000,"partial_rotary_factor":0.5,"original_max_position_embeddings":131072}}' \
|
| 73 |
-
--mem-fraction-static 0.85 \
|
| 74 |
--trust-remote-code \
|
| 75 |
--reasoning-parser auto \
|
| 76 |
--tool-call-parser auto \
|
|
@@ -78,7 +77,7 @@ docker run --rm --gpus all --ipc=host --shm-size 32g \
|
|
| 78 |
--port 30000
|
| 79 |
```
|
| 80 |
|
| 81 |
-
On 80GB cards (H100 / H800), scale out to `--tp 8`. The reasoning and tool-call parsers resolve automatically to `ling3` from the chat template; you can also set them explicitly with `--reasoning-parser ling3 --tool-call-parser ling3`.
|
| 82 |
|
| 83 |
**Client**
|
| 84 |
|
|
|
|
| 70 |
--tp 2 \
|
| 71 |
--context-length 262144 \
|
| 72 |
--json-model-override-args '{"rope_scaling":{"rope_type":"yarn","factor":2.0,"rope_theta":6000000,"partial_rotary_factor":0.5,"original_max_position_embeddings":131072}}' \
|
|
|
|
| 73 |
--trust-remote-code \
|
| 74 |
--reasoning-parser auto \
|
| 75 |
--tool-call-parser auto \
|
|
|
|
| 77 |
--port 30000
|
| 78 |
```
|
| 79 |
|
| 80 |
+
On 80GB cards (H100 / H800), scale out to `--tp 8 --ep 8` (plain `--tp 8` fails the 128×128 block-quantization divisibility check; `--ep 8` keeps experts whole per rank). The reasoning and tool-call parsers resolve automatically to `ling3` from the chat template; you can also set them explicitly with `--reasoning-parser ling3 --tool-call-parser ling3`.
|
| 81 |
|
| 82 |
**Client**
|
| 83 |
|