Instructions to use ibm-granite/granite-20b-code-base-8k with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use ibm-granite/granite-20b-code-base-8k with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="ibm-granite/granite-20b-code-base-8k")

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("ibm-granite/granite-20b-code-base-8k")
model = AutoModelForCausalLM.from_pretrained("ibm-granite/granite-20b-code-base-8k", device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use ibm-granite/granite-20b-code-base-8k with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "ibm-granite/granite-20b-code-base-8k"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ibm-granite/granite-20b-code-base-8k",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/ibm-granite/granite-20b-code-base-8k

SGLang

How to use ibm-granite/granite-20b-code-base-8k with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "ibm-granite/granite-20b-code-base-8k" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ibm-granite/granite-20b-code-base-8k",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "ibm-granite/granite-20b-code-base-8k" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ibm-granite/granite-20b-code-base-8k",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use ibm-granite/granite-20b-code-base-8k with Docker Model Runner:
```
docker model run hf.co/ibm-granite/granite-20b-code-base-8k
```

mayank-mishra commited on Aug 30, 2024

Commit

0a81cba

1 Parent(s): 862809c

update to r1.1

Browse files

Files changed (13) hide show

config.json +2 -2
generation_config.json +1 -1
model-00001-of-00009.safetensors +1 -1
model-00002-of-00009.safetensors +1 -1
model-00003-of-00009.safetensors +1 -1
model-00004-of-00009.safetensors +1 -1
model-00005-of-00009.safetensors +1 -1
model-00006-of-00009.safetensors +1 -1
model-00007-of-00009.safetensors +1 -1
model-00008-of-00009.safetensors +1 -1
model-00009-of-00009.safetensors +1 -1
tokenizer.json +1 -0
tokenizer_config.json +1 -1

config.json CHANGED Viewed

@@ -1,5 +1,5 @@
 {
-  "activation_function": "gelu_pytorch_tanh",
   "architectures": [
     "GPTBigCodeForCausalLM"
   ],
@@ -22,7 +22,7 @@
   "scale_attention_softmax_in_fp32": true,
   "scale_attn_weights": true,
   "torch_dtype": "bfloat16",
-  "transformers_version": "4.38.1",
   "use_cache": true,
   "vocab_size": 49152
 }

 {
+  "activation_function": "gelu",
   "architectures": [
     "GPTBigCodeForCausalLM"
   ],
   "scale_attention_softmax_in_fp32": true,
   "scale_attn_weights": true,
   "torch_dtype": "bfloat16",
+  "transformers_version": "4.41.2",
   "use_cache": true,
   "vocab_size": 49152
 }

generation_config.json CHANGED Viewed

@@ -3,5 +3,5 @@
   "bos_token_id": 0,
   "eos_token_id": 0,
   "pad_token_id": 0,
-  "transformers_version": "4.38.1"
 }

   "bos_token_id": 0,
   "eos_token_id": 0,
   "pad_token_id": 0,
+  "transformers_version": "4.41.2"
 }

model-00001-of-00009.safetensors CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:29a2622ae4c65993fbdb24731570b8e144b92083c743cd464740c53feedb07d6
 size 4952185256

 version https://git-lfs.github.com/spec/v1
+oid sha256:22b893e90be19cc42647f7d76caef1bd7101b6904df709b93472265fa6e41594
 size 4952185256

model-00002-of-00009.safetensors CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:4ceaa3f12eaa9c465094ca4a1a96a76af66d434f456b501572fdd47459f5ab36
 size 4930227792

 version https://git-lfs.github.com/spec/v1
+oid sha256:dc7f98d43d5aa8531381ffd2b7995008f1471a707a57becf638d089c549edec8
 size 4930227792

model-00003-of-00009.safetensors CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:68b205db105c140d016f85e80c2c27153ca557ad0095da542d34613d6ab27028
 size 4927118464

 version https://git-lfs.github.com/spec/v1
+oid sha256:1f56c8969bbdc7995f1c61123f4b63da7ff202a93e40fde98121ec9ccdc49556
 size 4927118464

model-00004-of-00009.safetensors CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:c4ad77a69ec76054eaf783c0f55070ae545ae1a6bf10d4457ec776696502cb94
 size 4930227864

 version https://git-lfs.github.com/spec/v1
+oid sha256:dd7060b8fac28950098ce792fc518a0a8af296f47e22709ccaad1ce8216591ea
 size 4930227864

model-00005-of-00009.safetensors CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:e9e5f77714b34e326b73a26684c5f98bfc883da65085fad9e28e25a3708194ab
 size 4927118464

 version https://git-lfs.github.com/spec/v1
+oid sha256:bbd071ecf07518991d9d82749d9c864a7e923eb82450048565e9121604f4cbe2
 size 4927118464

model-00006-of-00009.safetensors CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:c59ff4a90e1c82cb2788c2180cf37e0616b0920ff3fec480ac11e550d4ca6cdf
 size 4930227864

 version https://git-lfs.github.com/spec/v1
+oid sha256:b3a0e75fbd4db5862a20e23ce6ffbf69a4a28d59505085b8d82c1d61620cbf98
 size 4930227864

model-00007-of-00009.safetensors CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:f8cb945b3b369eee9e1020c3f3a6a89bc875ace3de1c69d0ac3b65dfa7ce990b
 size 4927118464

 version https://git-lfs.github.com/spec/v1
+oid sha256:8fee8bebaf2202e65cd8253898b317fc4a03a5a5cfb5bd6a01ccbf76a65d7c8b
 size 4927118464

model-00008-of-00009.safetensors CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:405e07efd511ce7c5b7250310026be2a998068d3441f33bf15968c4ce8c4d04a
 size 4930227864

 version https://git-lfs.github.com/spec/v1
+oid sha256:1fa8ce48dbb754a3e762d10d3590c65fb0eb30b6f28044f37ad184c335ff6916
 size 4930227864

model-00009-of-00009.safetensors CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:98f8e3576530d40a41001a6f0991cdd107e8d14147dfbbf1c5fa21f02eabf120
 size 679601176

 version https://git-lfs.github.com/spec/v1
+oid sha256:62f5a5189fb992fb52b72861f432dc19123bef17f78c42e1db8a89b208b4b5ab
 size 679601176

tokenizer.json CHANGED Viewed

@@ -206,6 +206,7 @@
     "end_of_word_suffix": null,
     "fuse_unk": false,
     "byte_fallback": false,
     "vocab": {
       "<|endoftext|>": 0,
       "<fim_prefix>": 1,

     "end_of_word_suffix": null,
     "fuse_unk": false,
     "byte_fallback": false,
+    "ignore_merges": false,
     "vocab": {
       "<|endoftext|>": 0,
       "<fim_prefix>": 1,

tokenizer_config.json CHANGED Viewed

@@ -178,7 +178,7 @@
   "bos_token": "<|endoftext|>",
   "clean_up_tokenization_spaces": true,
   "eos_token": "<|endoftext|>",
-  "model_max_length": 9223372036854775807,
   "pad_token": "<|endoftext|>",
   "padding_side": "left",
   "tokenizer_class": "GPT2Tokenizer",

   "bos_token": "<|endoftext|>",
   "clean_up_tokenization_spaces": true,
   "eos_token": "<|endoftext|>",
+  "model_max_length": 8192,
   "pad_token": "<|endoftext|>",
   "padding_side": "left",
   "tokenizer_class": "GPT2Tokenizer",