Instructions to use microsoft/Phi-3-mini-4k-instruct with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use microsoft/Phi-3-mini-4k-instruct with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="microsoft/Phi-3-mini-4k-instruct", trust_remote_code=True)
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("microsoft/Phi-3-mini-4k-instruct", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained("microsoft/Phi-3-mini-4k-instruct", trust_remote_code=True)
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Inference
Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use microsoft/Phi-3-mini-4k-instruct with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "microsoft/Phi-3-mini-4k-instruct"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "microsoft/Phi-3-mini-4k-instruct",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/microsoft/Phi-3-mini-4k-instruct

SGLang

How to use microsoft/Phi-3-mini-4k-instruct with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "microsoft/Phi-3-mini-4k-instruct" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "microsoft/Phi-3-mini-4k-instruct",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "microsoft/Phi-3-mini-4k-instruct" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "microsoft/Phi-3-mini-4k-instruct",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use microsoft/Phi-3-mini-4k-instruct with Docker Model Runner:
```
docker model run hf.co/microsoft/Phi-3-mini-4k-instruct
```

the new version(2024.7） raise ValueError(f"`rope_scaling`'s type field must be one of ['su', 'yarn'], got {rope_scaling_type}")

#82

by YuanTang96 - opened Jul 3, 2024

Discussion

YuanTang96

Jul 3, 2024

Thanks for updating new version of phi-3-mini! The improvement of new version is so outstanding！！

When I replace the new version(2024.7) to the old version(2024.4)， there is a new error that "raise ValueError(f"rope_scaling's type field must be one of ['su', 'yarn'], got {rope_scaling_type}")"

detail in the figure

I check the difference from "Phi-3-mini-4k-instruct/configuration_phi3.py" between the new version(2024.7) and the old version(2024.4). The code of https://huggingface.co/microsoft/Phi-3-mini-4k-instruct/blob/c1358f8a35e6d2af81890deffbbfa575b978c62f/configuration_phi3.py#L204 is updated.

I also check the the transformers（4.41.2）https://github.com/huggingface/transformers/blob/f91c16d270e5e3ff32fdb32ccf286d05c03dfa66/src/transformers/models/phi3/configuration_phi3.py#L184. The transformers（4.41.2） is just same to the old version(2024.4).

So, maybe, the transformers should be updated for the new version(2024.7)

Look for your reply!!

gargamit

Microsoft org Jul 3, 2024

@epicXiaoli it seems like that you are using the https://huggingface.co/microsoft/Phi-3-mini-128k-instruct model. Please ensure that you have set the trust_remote_code parameter to True. We have the PR (https://github.com/huggingface/transformers/pull/31436) to address this issue. Once the changes are incorporated into the official transformers' version, it won't be necessary anymore.

nguyenbh changed discussion status to closed Jul 9, 2024

xinyuf

Jul 17, 2024

hi I am facing the same issue from phi3forsequenceclassification class and for this class " The argument trust_remote_code is to be used with Auto classes. It has no effect here and is ignored." since the pull request has not been completed yet, for now do you think it makes sense to use "yarn" for phi3-mini-128k-instruct?

Shivashankarrs

Jul 18, 2024

I am facing the same issue as well.

arunnuveai

Nov 13, 2024

•

edited Jan 8, 2025

same here also,has any one resolved it :

when i called the microsoft/Phi-3-mini-128k-instruct model using HuggingFaceInferenceAPI it has worked but when i called the model using HuggingFaceLLM.It throws this error:
raise ValueError(f"rope_scaling's type field must be one of ['su', 'yarn'], got {rope_scaling_type}")

nguyenbh

Microsoft org Nov 13, 2024

@arunnuveai Thank you for your interest in Phi models.
I would recommend you to use Phi-3.5-Mini https://huggingface.co/microsoft/Phi-3.5-mini-instruct

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment