Instructions to use RtaForge/Anvaya-Rabbit-2.7B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use RtaForge/Anvaya-Rabbit-2.7B with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="RtaForge/Anvaya-Rabbit-2.7B", trust_remote_code=True)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained("RtaForge/Anvaya-Rabbit-2.7B", trust_remote_code=True, device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use RtaForge/Anvaya-Rabbit-2.7B with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "RtaForge/Anvaya-Rabbit-2.7B"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "RtaForge/Anvaya-Rabbit-2.7B",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/RtaForge/Anvaya-Rabbit-2.7B

SGLang

How to use RtaForge/Anvaya-Rabbit-2.7B with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "RtaForge/Anvaya-Rabbit-2.7B" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "RtaForge/Anvaya-Rabbit-2.7B",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "RtaForge/Anvaya-Rabbit-2.7B" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "RtaForge/Anvaya-Rabbit-2.7B",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use RtaForge/Anvaya-Rabbit-2.7B with Docker Model Runner:
```
docker model run hf.co/RtaForge/Anvaya-Rabbit-2.7B
```

tvastr commited on May 2

Commit

07bed52

verified ·

1 Parent(s): fd1efe6

Golden Checkpoint: Step 1397 (Logic Giant Saturation)

Browse files

Files changed (2) hide show

config.json +15 -0
pytorch_model.bin +3 -0

config.json ADDED Viewed

	@@ -0,0 +1,15 @@

+{
+    "nacre_tip": "8586164c7d4b4a88e325b9986570f0bf296880586df545c8c445092b5008014a",
+    "step": 1397,
+    "hash": "8586164c7d4b4a88e325b9986570f0bf296880586df545c8c445092b5008014a",
+    "weights_file": "/data/rabbit/scholar-efs/pearl/weights_step1397.pt",
+    "checkpoint_digest": "a260f132fb7c74b059178d273c14a2e7258e6c100c5a2f8b57026a39643a068e",
+    "checkpoint_layout": "full",
+    "base_weights_file": "/data/rabbit/scholar-efs/pearl/weights_step0.pt",
+    "surface_file": "",
+    "tokenizer_id": "EleutherAI/gpt-neox-20b",
+    "tokenizer_hash": "",
+    "schema_version": "2.0",
+    "gurukul_version": "phase2_hardened",
+    "stage_id": "stage00_sanskrit"
+}

pytorch_model.bin ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:a260f132fb7c74b059178d273c14a2e7258e6c100c5a2f8b57026a39643a068e
+size 12264396835