Instructions to use LinkSoul/Chinese-Llama-2-7b with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use LinkSoul/Chinese-Llama-2-7b with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="LinkSoul/Chinese-Llama-2-7b")

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("LinkSoul/Chinese-Llama-2-7b")
model = AutoModelForCausalLM.from_pretrained("LinkSoul/Chinese-Llama-2-7b", device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use LinkSoul/Chinese-Llama-2-7b with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "LinkSoul/Chinese-Llama-2-7b"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "LinkSoul/Chinese-Llama-2-7b",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/LinkSoul/Chinese-Llama-2-7b

SGLang

How to use LinkSoul/Chinese-Llama-2-7b with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "LinkSoul/Chinese-Llama-2-7b" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "LinkSoul/Chinese-Llama-2-7b",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "LinkSoul/Chinese-Llama-2-7b" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "LinkSoul/Chinese-Llama-2-7b",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use LinkSoul/Chinese-Llama-2-7b with Docker Model Runner:
```
docker model run hf.co/LinkSoul/Chinese-Llama-2-7b
```

用text-generation-webui加载模型后，速度很慢。

by xxxzsgxxx - opened Jul 28, 2023

Discussion

xxxzsgxxx

Jul 28, 2023

2023-07-28 18:13:19 INFO:Loading LinkSoul_Chinese-Llama-2-7b...
Loading checkpoint shards: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:25<00:00, 8.38s/it]
2023-07-28 18:13:45 WARNING:models/LinkSoul_Chinese-Llama-2-7b/tokenizer_config.json is different from the original LlamaTokenizer file. It is either customized or outdated.
2023-07-28 18:13:45 WARNING:models/LinkSoul_Chinese-Llama-2-7b/special_tokens_map.json is different from the original LlamaTokenizer file. It is either customized or outdated.
2023-07-28 18:13:45 INFO:Loaded the model in 25.91 seconds.

/Users/zsg/AI/oobabooga_macos/installer_files/env/lib/python3.10/site-packages/transformers/generation/utils.py:1270: UserWarning: You have modified the pretrained model configuration to control generation. This is a deprecated strategy to control generation and will be removed soon, in a future version. Please use a generation configuration file (see https://huggingface.co/docs/transformers/main_classes/text_generation )
warnings.warn(

shiyemin2

Jul 28, 2023

由于我们没有修改tokenizer，所以在输入、输出效率上是不如做了tokenizer修改的版本的

shiyemin2 changed discussion status to closed Jul 28, 2023

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment