Instructions to use RedHatAI/Kimi-K2.6-NVFP4 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use RedHatAI/Kimi-K2.6-NVFP4 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="RedHatAI/Kimi-K2.6-NVFP4", trust_remote_code=True)
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModel

processor = AutoProcessor.from_pretrained("RedHatAI/Kimi-K2.6-NVFP4", trust_remote_code=True)
model = AutoModel.from_pretrained("RedHatAI/Kimi-K2.6-NVFP4", trust_remote_code=True, device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use RedHatAI/Kimi-K2.6-NVFP4 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "RedHatAI/Kimi-K2.6-NVFP4"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "RedHatAI/Kimi-K2.6-NVFP4",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/RedHatAI/Kimi-K2.6-NVFP4

SGLang

How to use RedHatAI/Kimi-K2.6-NVFP4 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "RedHatAI/Kimi-K2.6-NVFP4" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "RedHatAI/Kimi-K2.6-NVFP4",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "RedHatAI/Kimi-K2.6-NVFP4" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "RedHatAI/Kimi-K2.6-NVFP4",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use RedHatAI/Kimi-K2.6-NVFP4 with Docker Model Runner:
```
docker model run hf.co/RedHatAI/Kimi-K2.6-NVFP4
```

nm-research commited on Jul 17

Commit

47fa8d9

verified ·

1 Parent(s): 50fb20e

Update BFCL v4 eval results (fresh run on vLLM 0.24.0, corrupted categories re-collected)

Browse files

Files changed (1) hide show

every_eval_ever/bfcl_v4.json +42 -42

every_eval_ever/bfcl_v4.json CHANGED Viewed

@@ -1,12 +1,12 @@
 {
   "schema_version": "0.2.2",
-  "evaluation_id": "bfcl_v4/RedHatAI/Kimi-K2.6-NVFP4/1782913700",
-  "retrieved_timestamp": "1782913700",
   "source_metadata": {
     "source_name": "BFCL",
     "source_type": "evaluation_run",
     "source_organization_name": "RedHatAI",
-    "evaluator_relationship": "third_party",
     "additional_details": {
       "benchmark_version": "v4"
     }
@@ -21,11 +21,11 @@
     "developer": "RedHatAI",
     "inference_engine": {
       "name": "vllm",
-      "version": "0.22.1"
     },
     "additional_details": {
       "profile": "kimi_nvfp4",
-      "evaluation_batch": "20260629T171225Z"
     }
   },
   "evaluation_results": [
@@ -36,7 +36,7 @@
         "source_type": "other",
         "additional_details": {
           "category": "non_live",
-          "source_csv": "/home/shubhra/kimik2.6_evals/runs/bfcl/20260629T171225Z_kimi_nvfp4_non_live/project_root/score/data_non_live.csv"
         }
       },
       "metric_config": {
@@ -47,20 +47,20 @@
         "max_score": 1.0
       },
       "score_details": {
-        "score": 0.8648,
         "details": {
           "Rank": "1",
           "Model": "RedHatAI/Kimi-K2.6-NVFP4 (FC, local vLLM)",
-          "Non-Live Overall Acc": "86.48%",
-          "AST Summary": "86.48%",
-          "Simple AST": "72.42%",
-          "Python Simple AST": "94.25%",
-          "Java Simple AST": "63.00%",
-          "JavaScript Simple AST": "60.00%",
-          "Multiple AST": "92.50%",
-          "Parallel AST": "91.50%",
-          "Parallel Multiple AST": "89.50%",
-          "Irrelevance Detection": "90.00%"
         }
       },
       "generation_config": {
@@ -83,7 +83,7 @@
         "source_type": "other",
         "additional_details": {
           "category": "live",
-          "source_csv": "/home/shubhra/kimik2.6_evals/runs/bfcl/20260629T171225Z_kimi_nvfp4_live/project_root/score/data_live.csv"
         }
       },
       "metric_config": {
@@ -94,17 +94,17 @@
         "max_score": 1.0
       },
       "score_details": {
-        "score": 0.7898000000000001,
         "details": {
           "Rank": "1",
           "Model": "RedHatAI/Kimi-K2.6-NVFP4 (FC, local vLLM)",
-          "Live Overall Acc": "78.98%",
-          "AST Summary": "78.98%",
-          "Python Simple AST": "86.82%",
-          "Python Multiple AST": "77.02%",
-          "Python Parallel AST": "81.25%",
-          "Python Parallel Multiple AST": "79.17%",
-          "Irrelevance Detection": "85.29%",
           "Relevance Detection": "56.25%"
         }
       },
@@ -128,7 +128,7 @@
         "source_type": "other",
         "additional_details": {
           "category": "multi_turn",
-          "source_csv": "/home/shubhra/kimik2.6_evals/runs/bfcl/20260629T171225Z_kimi_nvfp4_multi_turn/project_root/score/data_multi_turn.csv"
         }
       },
       "metric_config": {
@@ -144,8 +144,8 @@
           "Rank": "1",
           "Model": "RedHatAI/Kimi-K2.6-NVFP4 (FC, local vLLM)",
           "Multi Turn Overall Acc": "63.75%",
-          "Base": "71.50%",
-          "Miss Func": "64.00%",
           "Miss Param": "59.00%",
           "Long Context": "60.50%"
         }
@@ -170,7 +170,7 @@
         "source_type": "other",
         "additional_details": {
           "category": "memory",
-          "source_csv": "/home/shubhra/kimik2.6_evals/runs/bfcl/20260629T171225Z_kimi_nvfp4_memory/project_root/score/data_agentic.csv"
         }
       },
       "metric_config": {
@@ -181,17 +181,17 @@
         "max_score": 1.0
       },
       "score_details": {
-        "score": 0.6495000000000001,
         "details": {
           "Rank": "1",
           "Model": "RedHatAI/Kimi-K2.6-NVFP4 (FC, local vLLM)",
-          "Agentic Overall Acc": "32.47%",
           "Web Search Summary": "N/A",
           "Web Search Base": "N/A",
           "Web Search No Snippet": "N/A",
-          "Memory Summary": "64.95%",
-          "Memory KV": "56.77%",
-          "Memory Vector": "65.16%",
           "Memory Recursive Summarization": "72.90%"
         }
       },
@@ -215,7 +215,7 @@
         "source_type": "other",
         "additional_details": {
           "category": "web_search",
-          "source_csv": "/home/shubhra/kimik2.6_evals/runs/bfcl/20260629T171225Z_kimi_nvfp4_web_search/project_root/score/data_agentic.csv"
         }
       },
       "metric_config": {
@@ -226,14 +226,14 @@
         "max_score": 1.0
       },
       "score_details": {
-        "score": 0.42,
         "details": {
           "Rank": "1",
           "Model": "RedHatAI/Kimi-K2.6-NVFP4 (FC, local vLLM)",
-          "Agentic Overall Acc": "21.00%",
-          "Web Search Summary": "42.00%",
-          "Web Search Base": "53.00%",
-          "Web Search No Snippet": "31.00%",
           "Memory Summary": "N/A",
           "Memory KV": "N/A",
           "Memory Vector": "N/A",
@@ -254,4 +254,4 @@
       }
     }
   ]
-}

 {
   "schema_version": "0.2.2",
+  "evaluation_id": "bfcl_v4/RedHatAI/Kimi-K2.6-NVFP4/1784246968.2835383",
+  "retrieved_timestamp": "1784246968.2835383",
   "source_metadata": {
     "source_name": "BFCL",
     "source_type": "evaluation_run",
     "source_organization_name": "RedHatAI",
+    "evaluator_relationship": "first_party",
     "additional_details": {
       "benchmark_version": "v4"
     }
     "developer": "RedHatAI",
     "inference_engine": {
       "name": "vllm",
+      "version": "0.24.0"
     },
     "additional_details": {
       "profile": "kimi_nvfp4",
+      "evaluation_batch": "20260716"
     }
   },
   "evaluation_results": [
         "source_type": "other",
         "additional_details": {
           "category": "non_live",
+          "source_csv": "/home/shubhra/runs/bfcl/20260716T145522Z_kimi_nvfp4_non_live/project_root/score/data_non_live.csv"
         }
       },
       "metric_config": {
         "max_score": 1.0
       },
       "score_details": {
+        "score": 0.8644,
         "details": {
           "Rank": "1",
           "Model": "RedHatAI/Kimi-K2.6-NVFP4 (FC, local vLLM)",
+          "Non-Live Overall Acc": "86.44%",
+          "AST Summary": "86.44%",
+          "Simple AST": "70.25%",
+          "Python Simple AST": "94.75%",
+          "Java Simple AST": "60.00%",
+          "JavaScript Simple AST": "56.00%",
+          "Multiple AST": "93.00%",
+          "Parallel AST": "92.00%",
+          "Parallel Multiple AST": "90.50%",
+          "Irrelevance Detection": "88.33%"
         }
       },
       "generation_config": {
         "source_type": "other",
         "additional_details": {
           "category": "live",
+          "source_csv": "/home/shubhra/runs/bfcl/20260714T003915Z_kimi_nvfp4_live/project_root/score/data_live.csv"
         }
       },
       "metric_config": {
         "max_score": 1.0
       },
       "score_details": {
+        "score": 0.7831,
         "details": {
           "Rank": "1",
           "Model": "RedHatAI/Kimi-K2.6-NVFP4 (FC, local vLLM)",
+          "Live Overall Acc": "78.31%",
+          "AST Summary": "78.31%",
+          "Python Simple AST": "83.33%",
+          "Python Multiple AST": "76.73%",
+          "Python Parallel AST": "93.75%",
+          "Python Parallel Multiple AST": "83.33%",
+          "Irrelevance Detection": "84.39%",
           "Relevance Detection": "56.25%"
         }
       },
         "source_type": "other",
         "additional_details": {
           "category": "multi_turn",
+          "source_csv": "/home/shubhra/runs/bfcl/20260716T192814Z_kimi_nvfp4_multi_turn/project_root/score/data_multi_turn.csv"
         }
       },
       "metric_config": {
           "Rank": "1",
           "Model": "RedHatAI/Kimi-K2.6-NVFP4 (FC, local vLLM)",
           "Multi Turn Overall Acc": "63.75%",
+          "Base": "70.50%",
+          "Miss Func": "65.00%",
           "Miss Param": "59.00%",
           "Long Context": "60.50%"
         }
         "source_type": "other",
         "additional_details": {
           "category": "memory",
+          "source_csv": "/home/shubhra/runs/bfcl/20260716T174417Z_kimi_nvfp4_memory/project_root/score/data_agentic.csv"
         }
       },
       "metric_config": {
         "max_score": 1.0
       },
       "score_details": {
+        "score": 0.6323,
         "details": {
           "Rank": "1",
           "Model": "RedHatAI/Kimi-K2.6-NVFP4 (FC, local vLLM)",
+          "Agentic Overall Acc": "31.61%",
           "Web Search Summary": "N/A",
           "Web Search Base": "N/A",
           "Web Search No Snippet": "N/A",
+          "Memory Summary": "63.23%",
+          "Memory KV": "50.97%",
+          "Memory Vector": "65.81%",
           "Memory Recursive Summarization": "72.90%"
         }
       },
         "source_type": "other",
         "additional_details": {
           "category": "web_search",
+          "source_csv": "/home/shubhra/runs/bfcl/20260716T164559Z_kimi_nvfp4_web_search/project_root/score/data_agentic.csv"
         }
       },
       "metric_config": {
         "max_score": 1.0
       },
       "score_details": {
+        "score": 0.13,
         "details": {
           "Rank": "1",
           "Model": "RedHatAI/Kimi-K2.6-NVFP4 (FC, local vLLM)",
+          "Agentic Overall Acc": "6.50%",
+          "Web Search Summary": "13.00%",
+          "Web Search Base": "11.00%",
+          "Web Search No Snippet": "15.00%",
           "Memory Summary": "N/A",
           "Memory KV": "N/A",
           "Memory Vector": "N/A",
       }
     }
   ]
+}