Instructions to use RedHatAI/Kimi-K2.6-NVFP4 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use RedHatAI/Kimi-K2.6-NVFP4 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="RedHatAI/Kimi-K2.6-NVFP4", trust_remote_code=True)
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModel

processor = AutoProcessor.from_pretrained("RedHatAI/Kimi-K2.6-NVFP4", trust_remote_code=True)
model = AutoModel.from_pretrained("RedHatAI/Kimi-K2.6-NVFP4", trust_remote_code=True, device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use RedHatAI/Kimi-K2.6-NVFP4 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "RedHatAI/Kimi-K2.6-NVFP4"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "RedHatAI/Kimi-K2.6-NVFP4",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/RedHatAI/Kimi-K2.6-NVFP4

SGLang

How to use RedHatAI/Kimi-K2.6-NVFP4 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "RedHatAI/Kimi-K2.6-NVFP4" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "RedHatAI/Kimi-K2.6-NVFP4",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "RedHatAI/Kimi-K2.6-NVFP4" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "RedHatAI/Kimi-K2.6-NVFP4",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use RedHatAI/Kimi-K2.6-NVFP4 with Docker Model Runner:
```
docker model run hf.co/RedHatAI/Kimi-K2.6-NVFP4
```

bdellabe commited on Apr 30

Commit

d620f6d

verified ·

1 Parent(s): 166852a

Add files using upload-large-folder tool

Browse files

Files changed (38) hide show

model-00007-of-00119.safetensors +3 -0
model-00013-of-00119.safetensors +3 -0
model-00015-of-00119.safetensors +3 -0
model-00016-of-00119.safetensors +3 -0
model-00018-of-00119.safetensors +3 -0
model-00024-of-00119.safetensors +3 -0
model-00025-of-00119.safetensors +3 -0
model-00029-of-00119.safetensors +3 -0
model-00030-of-00119.safetensors +3 -0
model-00032-of-00119.safetensors +3 -0
model-00035-of-00119.safetensors +3 -0
model-00039-of-00119.safetensors +3 -0
model-00043-of-00119.safetensors +3 -0
model-00046-of-00119.safetensors +3 -0
model-00047-of-00119.safetensors +3 -0
model-00052-of-00119.safetensors +3 -0
model-00057-of-00119.safetensors +3 -0
model-00059-of-00119.safetensors +3 -0
model-00060-of-00119.safetensors +3 -0
model-00061-of-00119.safetensors +3 -0
model-00071-of-00119.safetensors +3 -0
model-00072-of-00119.safetensors +3 -0
model-00076-of-00119.safetensors +3 -0
model-00083-of-00119.safetensors +3 -0
model-00084-of-00119.safetensors +3 -0
model-00086-of-00119.safetensors +3 -0
model-00092-of-00119.safetensors +3 -0
model-00094-of-00119.safetensors +3 -0
model-00098-of-00119.safetensors +3 -0
model-00099-of-00119.safetensors +3 -0
model-00103-of-00119.safetensors +3 -0
model-00105-of-00119.safetensors +3 -0
model-00107-of-00119.safetensors +3 -0
model-00110-of-00119.safetensors +3 -0
model-00112-of-00119.safetensors +3 -0
model-00114-of-00119.safetensors +3 -0
model-00115-of-00119.safetensors +3 -0
model-00119-of-00119.safetensors +3 -0

model-00007-of-00119.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:e705385cd06344a9abdfb73e86bead73ab09cc1f461217cace9fe67e5dd99a82
+size 4997480128

model-00013-of-00119.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:370e5f3052d8059a28d9762de70244118763d827c9fa102ed50beeb5de8214b0
+size 4997479808

model-00015-of-00119.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:12437afc6b2a49532ddbf551aade270ef0b27846f4fde80e5c97dc1bd504457b
+size 4997479808

model-00016-of-00119.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:0a2da60e90592681a2cecf5ae932244f7a29eecd6d4aeb9aca26f855b20b1446
+size 4996149224

model-00018-of-00119.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:223a18d229aef5e63057873ad93cf7a50afd4292d22f33da8c3fcb6db258e939
+size 4996149224

model-00024-of-00119.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:4c8c529ef5924a9223d39fd7be8cc652f789b30a0f0109e0630ef3d2ef7470b2
+size 4996151648

model-00025-of-00119.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:f6172f9b2f0492e53f54e9c9883780aa81cbb621818a24564b69f07d79a90277
+size 4997482144

model-00029-of-00119.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:35b53a784b63a056877b4de2c2af95f3b2c269904b84147af0cabaa3730b5ff7
+size 4997482144

model-00030-of-00119.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:fe138fcb0c3afc70df37ff56c533f26ea8c0e1ccb91d9044e767df4c3ffab394
+size 4993428288

model-00032-of-00119.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:79184b7e1f0f2ed8ca65505a0d6891d93ffa3903f7c870b03373db5d6fc7d06a
+size 4997483416

model-00035-of-00119.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:69e5d8de18948c536f076b67f67cd54c941dce8d2f6aed5db57d46f691f7b11f
+size 4996150584

model-00039-of-00119.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:d371903906a7f9b159bfe9155e01a6ce81bb1abdfda27125fbdaac27582193b1
+size 4996150824

model-00043-of-00119.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:f533bb6352dcc0f23b82e02ef224984cbf3f3e0b2950b28af562fc2368cf3852
+size 4996151064

model-00046-of-00119.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:b8a6346c4d4f838017fd97bd56ea1a35f2954d19e34f5d9d6fc13c79543ba937
+size 4997482480

model-00047-of-00119.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:e06cc033a181d45c0b5bbb59a257aee9ea2dbb989f06d508f01512dff61f65ef
+size 4996151304

model-00052-of-00119.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:e504319ebd827b6833247bd7251ac80efc7b3c0b276f62eed0591d01c91ce43c
+size 4997482144

model-00057-of-00119.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:5b6ba5c690da8a13e3c18ed2767ff998effd1a8840e9f8c6f96a7d5149fa71ba
+size 4996151640

model-00059-of-00119.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:81d5a24e462958bca9bcc82bbe7121909ae4e7d914d3fa7325e60d641f6d12ee
+size 4996151640

model-00060-of-00119.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:5a792564ff21186a05ed41b3a461c3937d62b7231a576cfef5d2a313ddc92aae
+size 4997482144

model-00061-of-00119.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:73232c0992eaebd2a621fa3881357804fb3f8e13f08faf491e05581b2f63e524
+size 4996151640

model-00071-of-00119.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:9da849e7046cf0f2a2f98a0176ba64e583f5e95bad4277f10addb8bb2686c70e
+size 4997483448

model-00072-of-00119.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:91cb39437b3deaf0bdf9e20c2a5b3e9cc6dd943b83a71cfd4655f2a0742f9da2
+size 4996150336

model-00076-of-00119.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:51f9b8135c41a2426a7d240bd3f5d47133a3dd23c1956c5db3de3f5a2fb6acd5
+size 4996150688

model-00083-of-00119.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:ed94d418b0ded73cf5fa4f6a0d0165dba40b4f96779b34d42e649f972ffe60cc
+size 4997482616

model-00084-of-00119.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:b1ccef58c670d8b75c1f19c811661fe45daa67591e521923a5054b9ea38bb1ef
+size 4996151168

model-00086-of-00119.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:d2d3ea8ff832356418d27478cf2816cdae87e311b6c252a0149cc328c89cd56c
+size 4996151288

model-00092-of-00119.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:261ea09994f53c9c4f3621a7bb6acf933b27c466728802a7d04eea72aa37a68f
+size 4996151648

model-00094-of-00119.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:4b4978888796cb9d0c080b52342712e58194923ecf034a738e0b197f3a9b19ab
+size 4996151648

model-00098-of-00119.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:c89fd0b4bd5032d492504a6e3575b704fb0cc371cfc738b6e0ac20cf98c2aad9
+size 4996151648

model-00099-of-00119.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:ee78093c8fe4e66b66836209f6169f060d2524ffb9665e17c43ed28ea4c7ef55
+size 4997482144

model-00103-of-00119.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:2d9d99cdfa5368f4b9d483b98e3f628248711bf9918f13bf0dfbd35619955523
+size 4997482144

model-00105-of-00119.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:3e8ff1466401e22a01120e1c5d18ee022d7ca84e55c8ce4a2abf13f68e3b3b9f
+size 4997482144

model-00107-of-00119.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:268b07fbadd79bb3235f97f2711d915f2bdc4de91f7f34ef19e2db894afb184d
+size 4997482144

model-00110-of-00119.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:81fb7ec5e8df79011a14dd77c70240f27bc415809a793995dc5e2fff9efd4042
+size 4913074904

model-00112-of-00119.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:1a0cacb17bb99480a21dfc2fb60d7a749239597af9425ea238e691d191d4046f
+size 4997483328

model-00114-of-00119.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:fc3cc67f9346cb7305667905f257b1385011f952194549e8e7cdf746366381af
+size 4997483152

model-00115-of-00119.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:52f9b3c953aecec43b42401f439762a848d74ea2248e86f864dfb995ea0617f7
+size 4996150632

model-00119-of-00119.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:b94fb48e1c4ff5d8e60d8b93d68a326a33cd86b0120eb33cdcb0a59d568eadda
+size 2348810384