LightOnOCR-3-4B (GGUF)

GGUF quantizations of lightonai/LightOnOCR-3-4B — LightOn's most accurate LightOnOCR-3 model (Qwen3.5 vision-language architecture): end-to-end OCR with layout and visual understanding — optimized for local inference with llama.cpp and Ollama, with its vision projector. All quants use an importance matrix.

Quick Start

With Ollama

ollama run richardyoung/lightonocr-3-4b
# or straight from Hugging Face
ollama run hf.co/richardyoung/LightOnOCR-3-4B-GGUF:Q4_K_M

With llama.cpp

huggingface-cli download richardyoung/LightOnOCR-3-4B-GGUF \
    --include "*Q4_K_M*" "mmproj*" --local-dir ./models

./llama-mtmd-cli -m ./models/LightOnOCR-3-4B-Q4_K_M.gguf \
    --mmproj ./models/mmproj-LightOnOCR-3-4B-F16.gguf \
    --image page.png -p "grounding"

Prompts: send the image with an empty prompt for plain transcription (Markdown), or with the prompt grounding to get every block with a label and a 0–1000 bounding box, image descriptions and chart data as tables. LightOn notes that other instructions are out of distribution. Recommended sampling: temperature 0.2, top_p 0.9 (set in the Ollama tags).

Why This Model?

  • Most accurate of the LightOnOCR-3 family — LightOn's recommendation for most OCR tasks
  • Two modes: plain transcription, or grounding with labels and bounding boxes for every block
  • Visual understanding: short descriptions for images, charts turned into tables of their data
  • Tables, receipts, forms, multi-column layouts and math notation
  • Apache-2.0: research and commercial use

Use Cases

  • Digitizing scanned documents and PDFs locally
  • Layout-aware extraction with bounding boxes
  • Chart and table data extraction for RAG

Available Quantizations

File Size Use Case
LightOnOCR-3-4B-IQ3_M.gguf 2.2 GB Smallest
LightOnOCR-3-4B-IQ4_XS.gguf 2.5 GB Great quality-size balance
LightOnOCR-3-4B-Q4_K_M.gguf 2.7 GB Recommended — good balance
LightOnOCR-3-4B-Q5_K_M.gguf 3.1 GB Higher quality
LightOnOCR-3-4B-Q6_K.gguf 3.5 GB Very high quality
LightOnOCR-3-4B-Q8_0.gguf 4.5 GB Near-lossless
mmproj-LightOnOCR-3-4B-F16.gguf 0.67 GB Vision projector (required for images)
LightOnOCR-3-4B-imatrix.gguf – Importance matrix used for the quants (calibration_datav3)

Quant Quality

Each quant compared with the F16 model using llama-perplexity --kl-divergence on the wikitext-2 test set (40 × 512-token chunks). KL divergence measures how far the quant's next-token probabilities drift from F16; lower is better. Top-token match is how often the quant picks the same next token as F16. As a rough guide, a KLD under 0.01 is hard to tell from F16, and from about 0.05 differences start to show in longer outputs. These numbers cover the text decoder only, on English prose. For OCR accuracy, see Verified below.

Quant Size KL divergence Top-token match Perplexity (F16: 8.54)
IQ3_M 2.2 GB 0.0843 86.4% 9.31
IQ4_XS 2.5 GB 0.0278 91.7% 8.76
Q4_K_M 2.7 GB 0.0210 92.8% 8.67
Q5_K_M 3.1 GB 0.0077 95.8% 8.63
Q6_K 3.5 GB 0.0042 96.6% 8.57
Q8_0 4.5 GB 0.0007 98.5% 8.54

Verified

Tested through Ollama before upload (Q4_K_M and Q8_0) on a synthetic invoice with an empty prompt; Q4_K_M returned: 'INVOICE 4471 / Date: 2026-09-28 / Widgets x3 ....... $42.00 / TOTAL: $86.20'. Converted with --no-mtp: the upstream config declares a multi-token-prediction layer whose weights are not in the release.

Also Available

Other Models by richardyoung


Built & maintained by Richard Young · DeepNeuro

Downloads last month
2,555
GGUF
Model size
4B params
Architecture
qwen35
Hardware compatibility
Log In to add your hardware

3-bit

4-bit

5-bit

6-bit

8-bit

Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for richardyoung/LightOnOCR-3-4B-GGUF

Quantized
(16)
this model

Collection including richardyoung/LightOnOCR-3-4B-GGUF

Paper for richardyoung/LightOnOCR-3-4B-GGUF