padeoe commited on
Commit
6d22d66
·
verified ·
1 Parent(s): bf3e690

init commit

Browse files
.gitattributes CHANGED
@@ -33,3 +33,5 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
 
 
 
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
36
+ granite-docling-258M-q8_0.gguf filter=lfs diff=lfs merge=lfs -text
37
+ mmproj-granite-docling-258M-f16.gguf filter=lfs diff=lfs merge=lfs -text
README.md CHANGED
@@ -1,3 +1,62 @@
1
- ---
2
- license: apache-2.0
3
- ---
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: apache-2.0
3
+ library_name: gguf
4
+ pipeline_tag: image-text-to-text
5
+ base_model:
6
+ - ibm-granite/granite-docling-258M
7
+ base_model_relation: quantized
8
+ tags:
9
+ - gguf
10
+ - vision
11
+ - multimodal
12
+ - ocr
13
+ - document-understanding
14
+ language:
15
+ - en
16
+ ---
17
+
18
+ # Granite-Docling-258M-GGUF
19
+
20
+ GGUF format of [ibm-granite/granite-docling-258M](https://huggingface.co/ibm-granite/granite-docling-258M), a multimodal document OCR model that converts document images to Docling format.
21
+
22
+ Converted with [llama.cpp](https://github.com/ggml-org/llama.cpp).
23
+
24
+ ## Files
25
+
26
+ | File | Quant | Size | Note |
27
+ |------|-------|------|------|
28
+ | `granite-docling-258M-f16.gguf` | F16 | 317 MB | Full precision |
29
+ | `granite-docling-258M-q8_0.gguf` | Q8_0 | 170 MB | Recommended |
30
+ | `mmproj-granite-docling-258M-f16.gguf` | F16 | 182 MB | Vision encoder (required) |
31
+
32
+ ## Usage
33
+
34
+ ### CLI
35
+
36
+ ```bash
37
+ llama-mtmd-cli \
38
+ --model granite-docling-258M-q8_0.gguf \
39
+ --mmproj mmproj-granite-docling-258M-f16.gguf \
40
+ --image document.png \
41
+ --n-predict 4096 --ctx-size 8192 --temp 0.0 \
42
+ -p "Convert this page to docling."
43
+ ```
44
+
45
+ ### Server (OpenAI-compatible API)
46
+
47
+ ```bash
48
+ llama-server \
49
+ -m granite-docling-258M-q8_0.gguf \
50
+ --mmproj mmproj-granite-docling-258M-f16.gguf \
51
+ --ctx-size 8192 --special --jinja \
52
+ --host 0.0.0.0 --port 8080
53
+ ```
54
+
55
+ ## Benchmark (CPU only, Q8_0)
56
+
57
+ | CPU | Config | Long text (4096 tok) | Short text (50 tok) |
58
+ |-----|--------|---------------------|---------------------|
59
+ | EPYC 9654 (96C) | 192 inst x 1t | 1.73 img/s | 29.4 img/s |
60
+ | EPYC 9654 (16C) | 16 inst x 1t | 0.67 img/s | 8.68 img/s |
61
+
62
+ For this small model, **1 thread per instance with max instances = core count** gives best throughput.
granite-docling-258M-q8_0.gguf ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:22db509defbff356a07fe5cf2b74b42b6a9c03fbf21a9ad12a41fe2a4ef7bc90
3
+ size 177918240
mmproj-granite-docling-258M-f16.gguf ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:4b7b2c4ed3162bd9cb323668c188e2324838b8f8ca4f3f9767f16c4c600d43ad
3
+ size 190031712