Instructions to use chatpdflocal/llama3.1-8b-gguf with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use chatpdflocal/llama3.1-8b-gguf with Transformers:
# pip install -U transformers accelerate # Load model directly from transformers import AutoModel model = AutoModel.from_pretrained("chatpdflocal/llama3.1-8b-gguf", device_map="auto") - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use chatpdflocal/llama3.1-8b-gguf with llama.cpp:
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible server with a web UI: llama serve -hf chatpdflocal/llama3.1-8b-gguf:Q4_K_M # Run inference directly in the terminal: llama cli -hf chatpdflocal/llama3.1-8b-gguf:Q4_K_M
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama serve -hf chatpdflocal/llama3.1-8b-gguf:Q4_K_M # Run inference directly in the terminal: llama cli -hf chatpdflocal/llama3.1-8b-gguf:Q4_K_M
Build from source code
git clone https://github.com/ggml-org/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-app # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama serve -hf chatpdflocal/llama3.1-8b-gguf:Q4_K_M # Run inference directly in the terminal: ./build/bin/llama cli -hf chatpdflocal/llama3.1-8b-gguf:Q4_K_M
Use Docker
docker model run hf.co/chatpdflocal/llama3.1-8b-gguf:Q4_K_M
- Llama new
- LM Studio
- Jan
- Ollama
How to use chatpdflocal/llama3.1-8b-gguf with Ollama:
ollama run hf.co/chatpdflocal/llama3.1-8b-gguf:Q4_K_M
- Unsloth Desktop
- Docker Model Runner
How to use chatpdflocal/llama3.1-8b-gguf with Docker Model Runner:
docker model run hf.co/chatpdflocal/llama3.1-8b-gguf:Q4_K_M
- Lemonade
How to use chatpdflocal/llama3.1-8b-gguf with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull chatpdflocal/llama3.1-8b-gguf:Q4_K_M
Run and chat with the model
lemonade run user.llama3.1-8b-gguf-Q4_K_M
List all available models
lemonade list
- Atomic Chat
Unable to load model when used with `llama-cli`
Trying to run ggml-model-Q4_K_M.gguf on my Macbook M1 Pro 16GB and getting this error. I've spent some time searching for the possible answers online, but wasn't able to find anything helpful.
Here are some relevant logs:
llama_new_context_with_model: n_ctx = 131072
llama_new_context_with_model: n_batch = 2048
llama_new_context_with_model: n_ubatch = 512
llama_new_context_with_model: flash_attn = 0
llama_new_context_with_model: freq_base = 500000.0
llama_new_context_with_model: freq_scale = 1
ggml_metal_init: allocating
ggml_metal_init: found device: Apple M1 Pro
ggml_metal_init: picking default device: Apple M1 Pro
ggml_metal_init: using embedded metal library
ggml_metal_init: GPU name: Apple M1 Pro
ggml_metal_init: GPU family: MTLGPUFamilyApple7 (1007)
ggml_metal_init: GPU family: MTLGPUFamilyCommon3 (3003)
ggml_metal_init: GPU family: MTLGPUFamilyMetal3 (5001)
ggml_metal_init: simdgroup reduction support = true
ggml_metal_init: simdgroup matrix mul. support = true
ggml_metal_init: hasUnifiedMemory = true
ggml_metal_init: recommendedMaxWorkingSetSize = 11453.25 MB
llama_kv_cache_init: Metal KV buffer size = 16384.00 MiB
llama_new_context_with_model: KV self size = 16384.00 MiB, K (f16): 8192.00 MiB, V (f16): 8192.00 MiB
llama_new_context_with_model: CPU output buffer size = 0.49 MiB
ggml_backend_metal_buffer_type_alloc_buffer: error: failed to allocate buffer, size = 8480.02 MiB
ggml_gallocr_reserve_n: failed to allocate Metal buffer of size 8891928576
llama_new_context_with_model: failed to allocate compute buffers
ggml_metal_free: deallocating
llama_init_from_gpt_params: error: failed to create context with model '.llama/7B/ggml-model-Q4_K_M.gguf'
main: error: unable to load model
Thanks for your issue, we will check it right now.
Hi, dimanis.
We have tested in X86 and m2 Mac Pro, both are ok.
See your output log, maybe you can try to reduce your metal buffer size.
By the way, could you give the start command of your llama-cli?
@Vincent-Lee , sure, I'm running it with parameters from the quick start section of the llama.cpp repo:llama-cli -m /path/to/model/model_name.gguf -p "I believe the meaning of life is" -n 128
@Vincent-Lee Are you building llama.cpp from source? I'm running the brew version. Perhaps that could be the culprit?
@Vincent-Lee , sure, I'm running it with parameters from the quick start section of the llama.cpp repo:
llama-cli -m /path/to/model/model_name.gguf -p "I believe the meaning of life is" -n 128
@dimanis
I doubt the reason is that the context window was set too large.
-c, --ctx-size N size of the prompt context (default: 0, 0 = loaded from model). Here without set prompt context, the size of prompt context loaded from model was set to llama_new_context_with_model: n_ctx = 131072.
Could you add the "-c 2048" to the start command and try to load the model?
@Vincent-Lee Are you building llama.cpp from source? I'm running the brew version. Perhaps that could be the culprit?
yes, we have built from source.
@Vincent-Lee yes, I can confirm that adding -c 2048 to the start command solved the problem. Thanks!