Instructions to use 01-ai/Yi-34B-Chat with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use 01-ai/Yi-34B-Chat with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="01-ai/Yi-34B-Chat")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("01-ai/Yi-34B-Chat")
model = AutoModelForCausalLM.from_pretrained("01-ai/Yi-34B-Chat")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps

vLLM

How to use 01-ai/Yi-34B-Chat with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "01-ai/Yi-34B-Chat"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "01-ai/Yi-34B-Chat",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/01-ai/Yi-34B-Chat

SGLang

How to use 01-ai/Yi-34B-Chat with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "01-ai/Yi-34B-Chat" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "01-ai/Yi-34B-Chat",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "01-ai/Yi-34B-Chat" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "01-ai/Yi-34B-Chat",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use 01-ai/Yi-34B-Chat with Docker Model Runner:
```
docker model run hf.co/01-ai/Yi-34B-Chat
```

What the SFT data?

by Ede-CH - opened Nov 29, 2023

Discussion

Ede-CH

Nov 29, 2023

Hi,
What instruction data are you using?

markding

Feb 15, 2024

I have the same question. I found reference to COIG and to rm-static in the code, but there is no datasheet available showing exactly what this model was instruction-tuned on and how. FWIW, we're keeping track of LLM openness at https://opening-up-chatgpt.github.io and Yi 34B Chat is currently in the bottom 5 (out of >30 'open' instruction tuned models) by degrees of openness because so little of source code, training data, instruction tuning etc. is shared or documented.

Andriy

Mar 31, 2024

Please answer the question.

YShow

Apr 30, 2024

Please refer to the technical report：https://arxiv.org/abs/2403.04652

markding

Apr 30, 2024

•

edited Apr 30, 2024

That preprint describes SFT data selection methods and cites some work it takes inspiration from, but as far as I can see does not specify any of the actual training data sets used.

Our finetuning dataset consists of less than 10K multi-turn instruction-response dialog pairs, with each and every one of the entry constructed and polished over multiple iterations and from user feedback.

What are those dialog pairs, where were they sourced from, what was the user feedback?

lorinma

01-ai org Jun 3, 2024

The SFT data was manually created in-house.

lorinma changed discussion status to closed Jun 3, 2024

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment