Instructions to use microsoft/Phi-3-mini-4k-instruct-onnx-web with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use microsoft/Phi-3-mini-4k-instruct-onnx-web with Transformers.js:

// npm i @huggingface/transformers
import { pipeline } from '@huggingface/transformers';

// Allocate pipeline
const pipe = await pipeline('text-generation', 'microsoft/Phi-3-mini-4k-instruct-onnx-web');

Transformers

How to use microsoft/Phi-3-mini-4k-instruct-onnx-web with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="microsoft/Phi-3-mini-4k-instruct-onnx-web", trust_remote_code=True)
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("microsoft/Phi-3-mini-4k-instruct-onnx-web", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained("microsoft/Phi-3-mini-4k-instruct-onnx-web", trust_remote_code=True)
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps

vLLM

How to use microsoft/Phi-3-mini-4k-instruct-onnx-web with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "microsoft/Phi-3-mini-4k-instruct-onnx-web"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "microsoft/Phi-3-mini-4k-instruct-onnx-web",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/microsoft/Phi-3-mini-4k-instruct-onnx-web

SGLang

How to use microsoft/Phi-3-mini-4k-instruct-onnx-web with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "microsoft/Phi-3-mini-4k-instruct-onnx-web" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "microsoft/Phi-3-mini-4k-instruct-onnx-web",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "microsoft/Phi-3-mini-4k-instruct-onnx-web" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "microsoft/Phi-3-mini-4k-instruct-onnx-web",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use microsoft/Phi-3-mini-4k-instruct-onnx-web with Docker Model Runner:
```
docker model run hf.co/microsoft/Phi-3-mini-4k-instruct-onnx-web
```

Phi-3 Mini-4K-Instruct ONNX model for in-browser inference

Running Phi3-mini-4K entirely in the browser! Check out this demo.

This repository hosts the optimized Web version of ONNX Phi-3-mini-4k-instruct model to accelerate inference in the browser with ONNX Runtime Web.

The Phi-3-Mini-4K-Instruct is a 3.8B parameters, lightweight, state-of-the-art open model trained with the Phi-3 datasets that includes both synthetic data and the filtered publicly available websites data with a focus on high-quality and reasoning dense properties. When assessed against benchmarks testing common sense, language understanding, math, code, long context and logical reasoning, Phi-3 Mini-4K-Instruct showcased a robust and state-of-the-art performance among models with less than 13 billion parameters.

How to run

ONNX Runtime Web is a JavaScript library to enable web developers to deploy machine learning models directly in web browsers, offering multiple backends leveraging hardware acceleration. WebGPU backend is recommended to run Phi-3-mini efficiently.

Here is an E2E example for running this optimized Phi3-mini-4K for the web, with ONNX Runtime harnessing WebGPU.

Supported devices and browser with WebGPU: Chrome 113+ and Edge 113+ for Mac, Windows, ChromeOS, and Chrome 121+ for Android. Pls visit here for tracking WebGPU support in browsers

Performance Metrics

Performance vary between GPUs. The more powerful the GPU, the faster the speed. On a NVIDIA GeForce RTX 4090: ~42 tokens/second

Additional Details

To obtain other optimized Phi3-mini-4k ONNX models for server platforms, Windows, Linux, Mac desktops, and mobile, please visit Phi-3-mini-4k-instruct onnx model. The model differences in the web version compared to other versions:

the model is fp16 with int4 block quantization for weights
the 'logits' output is fp32
the model uses MHA instead of GQA
onnx and external data file need to stay below 2GB to be cacheable in chromium

To optimize a fine-tuned Phi3-mini-4k model to run with ONNX Runtime Web, please follow this Olive example. Olive is an easy-to-use model optimization tool for generating an optimized ONNX model to efficiently run with ONNX Runtime across platforms.

Model Description

Developed by: Microsoft
Model type: ONNX
Inference Language(s) (NLP): JavaScript
License: MIT
Model Description: This is the web version of the Phi-3 Mini-4K-Instruct model for ONNX Runtime inference.

Model Card Contact

guschmue, qining

Downloads last month: 224

Collection including microsoft/Phi-3-mini-4k-instruct-onnx-web

Phi-3

Collection

Phi-3 family of small language and multi-modal models. Language models are available in short- and long-context lengths. • 25 items • Updated Mar 2 • 580