vllm
patrickvonplaten commited on
Commit
ee17cba
·
verified ·
1 Parent(s): 427c008

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +28 -6
README.md CHANGED
@@ -75,7 +75,7 @@ or via a *local server* (see [vLLM](#vllm-recommended)):
75
  ```
76
  [[providers]]
77
  name = "vllm"
78
- api_base = "http://<your-host-url>:800/v1"
79
  reasoning_as_structured_content = true
80
  reasoning_field_name = "reasoning_content"
81
  [[models]]
@@ -115,17 +115,39 @@ to implement production-ready inference pipelines.
115
 
116
  **_Installation_**
117
 
118
- Please make sure to install latest vLLM:
 
 
 
119
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
120
  ```
121
- uv pip install -U vllm
122
  ```
123
 
124
- Alternatively you can also directly use the latest docker image [vllm/vllm-openai:latest](https://hub.docker.com/layers/vllm/vllm-openai/latest/):
 
 
 
125
 
 
126
  ```
127
- docker pull vllm/vllm-openai:latest
128
- docker run -it vllm/vllm-openai:latest
129
  ```
130
 
131
  Also make sure to have installed [`mistral_common >= 1.10.0`](https://github.com/mistralai/mistral-common/releases/tag/v1.10.0).
 
75
  ```
76
  [[providers]]
77
  name = "vllm"
78
+ api_base = "http://<your-host-url>:8000/v1"
79
  reasoning_as_structured_content = true
80
  reasoning_field_name = "reasoning_content"
81
  [[models]]
 
115
 
116
  **_Installation_**
117
 
118
+ > [!Tip]
119
+ > We recommend installing vLLM from our custom Docker image that has fixes for
120
+ > Tool Calling and Reasoning parsing in vLLM and uses the latest version of Transformers.
121
+ > We're working with the vLLM team to merge these fixes to vLLM's main as soon as possible.
122
 
123
+ **_Custom Docker_**
124
+
125
+ Make sure to use the following docker image [mistralllm/vllm-ms4:latest](https://hub.docker.com/repository/docker/mistralllm/vllm-ms4/latest/):
126
+
127
+ ```
128
+ docker pull mistralllm/vllm-ms4:latest
129
+ docker run -it mistralllm/vllm-ms4:latest
130
+ ```
131
+
132
+ **_Manual Install_**
133
+
134
+ If you prefer, you can also manually install `vllm` from this PR: [Add Mistral Guidance](https://github.com/vllm-project/vllm/pull/37081).
135
+
136
+ **Note**: It is likely that this PR will be split into smaller PRs and merged to `vllm` main in the coming 1-2 weeks (Stand: 16.03.2026).
137
+
138
+ 1. Git clone vLLM:
139
  ```
140
+ git clone --branch fix_mistral_parsing https://github.com/juliendenize/vllm.git
141
  ```
142
 
143
+ 2. Install with pre-compiled kernels
144
+ ```
145
+ VLLM_USE_PRECOMPILED=1 pip install --editable .
146
+ ```
147
 
148
+ 3. Make sure, `transformers` is installed from "main":
149
  ```
150
+ pip install git+https://github.com/huggingface/transformers.git
 
151
  ```
152
 
153
  Also make sure to have installed [`mistral_common >= 1.10.0`](https://github.com/mistralai/mistral-common/releases/tag/v1.10.0).