TFMC/imatrix-dataset-for-japanese-llm
Viewer • Updated • 239 • 104 • 35
How to use mmnga/Fugaku-LLM-13B-instruct-gguf with llama.cpp:
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible server with a web UI: llama serve -hf mmnga/Fugaku-LLM-13B-instruct-gguf:Q4_K_M # Run inference directly in the terminal: llama cli -hf mmnga/Fugaku-LLM-13B-instruct-gguf:Q4_K_M
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama serve -hf mmnga/Fugaku-LLM-13B-instruct-gguf:Q4_K_M # Run inference directly in the terminal: llama cli -hf mmnga/Fugaku-LLM-13B-instruct-gguf:Q4_K_M
git clone https://github.com/ggml-org/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-app # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama serve -hf mmnga/Fugaku-LLM-13B-instruct-gguf:Q4_K_M # Run inference directly in the terminal: ./build/bin/llama cli -hf mmnga/Fugaku-LLM-13B-instruct-gguf:Q4_K_M
docker model run hf.co/mmnga/Fugaku-LLM-13B-instruct-gguf:Q4_K_M
How to use mmnga/Fugaku-LLM-13B-instruct-gguf with Ollama:
ollama run hf.co/mmnga/Fugaku-LLM-13B-instruct-gguf:Q4_K_M
How to use mmnga/Fugaku-LLM-13B-instruct-gguf with Docker Model Runner:
docker model run hf.co/mmnga/Fugaku-LLM-13B-instruct-gguf:Q4_K_M
How to use mmnga/Fugaku-LLM-13B-instruct-gguf with Lemonade:
# Download Lemonade from https://lemonade-server.ai/ lemonade pull mmnga/Fugaku-LLM-13B-instruct-gguf:Q4_K_M
lemonade run user.Fugaku-LLM-13B-instruct-gguf-Q4_K_M
lemonade list
Fugaku-LLMさんが公開しているFugaku-LLM-13B-instructのggufフォーマット変換版です。
imatrixのデータはTFMC/imatrix-dataset-for-japanese-llmを使用して作成しました。
ご使用の前に必ず利用規約をご確認ください。
ご使用の際には利用規約に同意したものとされます。
Please be sure to check the terms of use before use. By using this service, you agree to the Terms of Use.
変換用スクリプトはこちら
git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
make -j
./main -m 'Fugaku-LLM-13B-instruct-Q4_0.gguf' -n 128 -p '以下は、タスクを説明する指示です。要求を適切に満たす応答を書きなさい。\n\n### 指示:\nスーパーコンピュータ「富岳」の名前の由来を教えてください。\n\n### 応答:\n'
1-bit
2-bit
3-bit
4-bit
5-bit
6-bit
8-bit