Update README.md
Browse files
README.md
CHANGED
|
@@ -75,7 +75,7 @@ or via a *local server* (see [vLLM](#vllm-recommended)):
|
|
| 75 |
```
|
| 76 |
[[providers]]
|
| 77 |
name = "vllm"
|
| 78 |
-
api_base = "http://<your-host-url>:
|
| 79 |
reasoning_as_structured_content = true
|
| 80 |
reasoning_field_name = "reasoning_content"
|
| 81 |
[[models]]
|
|
@@ -115,17 +115,39 @@ to implement production-ready inference pipelines.
|
|
| 115 |
|
| 116 |
**_Installation_**
|
| 117 |
|
| 118 |
-
|
|
|
|
|
|
|
|
|
|
| 119 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 120 |
```
|
| 121 |
-
|
| 122 |
```
|
| 123 |
|
| 124 |
-
|
|
|
|
|
|
|
|
|
|
| 125 |
|
|
|
|
| 126 |
```
|
| 127 |
-
|
| 128 |
-
docker run -it vllm/vllm-openai:latest
|
| 129 |
```
|
| 130 |
|
| 131 |
Also make sure to have installed [`mistral_common >= 1.10.0`](https://github.com/mistralai/mistral-common/releases/tag/v1.10.0).
|
|
|
|
| 75 |
```
|
| 76 |
[[providers]]
|
| 77 |
name = "vllm"
|
| 78 |
+
api_base = "http://<your-host-url>:8000/v1"
|
| 79 |
reasoning_as_structured_content = true
|
| 80 |
reasoning_field_name = "reasoning_content"
|
| 81 |
[[models]]
|
|
|
|
| 115 |
|
| 116 |
**_Installation_**
|
| 117 |
|
| 118 |
+
> [!Tip]
|
| 119 |
+
> We recommend installing vLLM from our custom Docker image that has fixes for
|
| 120 |
+
> Tool Calling and Reasoning parsing in vLLM and uses the latest version of Transformers.
|
| 121 |
+
> We're working with the vLLM team to merge these fixes to vLLM's main as soon as possible.
|
| 122 |
|
| 123 |
+
**_Custom Docker_**
|
| 124 |
+
|
| 125 |
+
Make sure to use the following docker image [mistralllm/vllm-ms4:latest](https://hub.docker.com/repository/docker/mistralllm/vllm-ms4/latest/):
|
| 126 |
+
|
| 127 |
+
```
|
| 128 |
+
docker pull mistralllm/vllm-ms4:latest
|
| 129 |
+
docker run -it mistralllm/vllm-ms4:latest
|
| 130 |
+
```
|
| 131 |
+
|
| 132 |
+
**_Manual Install_**
|
| 133 |
+
|
| 134 |
+
If you prefer, you can also manually install `vllm` from this PR: [Add Mistral Guidance](https://github.com/vllm-project/vllm/pull/37081).
|
| 135 |
+
|
| 136 |
+
**Note**: It is likely that this PR will be split into smaller PRs and merged to `vllm` main in the coming 1-2 weeks (Stand: 16.03.2026).
|
| 137 |
+
|
| 138 |
+
1. Git clone vLLM:
|
| 139 |
```
|
| 140 |
+
git clone --branch fix_mistral_parsing https://github.com/juliendenize/vllm.git
|
| 141 |
```
|
| 142 |
|
| 143 |
+
2. Install with pre-compiled kernels
|
| 144 |
+
```
|
| 145 |
+
VLLM_USE_PRECOMPILED=1 pip install --editable .
|
| 146 |
+
```
|
| 147 |
|
| 148 |
+
3. Make sure, `transformers` is installed from "main":
|
| 149 |
```
|
| 150 |
+
pip install git+https://github.com/huggingface/transformers.git
|
|
|
|
| 151 |
```
|
| 152 |
|
| 153 |
Also make sure to have installed [`mistral_common >= 1.10.0`](https://github.com/mistralai/mistral-common/releases/tag/v1.10.0).
|