Instructions to use LiAutoAD/Ristretto-3B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use LiAutoAD/Ristretto-3B with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="LiAutoAD/Ristretto-3B", trust_remote_code=True)
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoModel
model = AutoModel.from_pretrained("LiAutoAD/Ristretto-3B", trust_remote_code=True, dtype="auto")

Notebooks
Google Colab
Kaggle
Local Apps

vLLM

How to use LiAutoAD/Ristretto-3B with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "LiAutoAD/Ristretto-3B"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "LiAutoAD/Ristretto-3B",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/LiAutoAD/Ristretto-3B

SGLang

How to use LiAutoAD/Ristretto-3B with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "LiAutoAD/Ristretto-3B" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "LiAutoAD/Ristretto-3B",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "LiAutoAD/Ristretto-3B" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "LiAutoAD/Ristretto-3B",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use LiAutoAD/Ristretto-3B with Docker Model Runner:
```
docker model run hf.co/LiAutoAD/Ristretto-3B
```

What is the LLM backbone?

by niktheod - opened 26 days ago

Discussion

niktheod

26 days ago

Hi, what is the LLM backbone for Ristretto. In transformers the LLM is Qwen2ForCausalLM, however its size is around 3.4B parameters. There isn't a model of this size in the Qwen2 family. Can you clarify what backbone was used?

mjl1206

LiAutoAD org 21 days ago

•

edited 21 days ago

Hi, what is the LLM backbone for Ristretto. In transformers the LLM is Qwen2ForCausalLM, however its size is around 3.4B parameters. There isn't a model of this size in the Qwen2 family. Can you clarify what backbone was used?

As shown in the Model Card, our LLM utilizes the Qwen2.5-3B architecture. Since we have set tie_word_embeddings to False, the LM head weights are not shared with the embedding layer, resulting in an additional parameter count of approximately 300M.

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment