Model Card for phucdu/qwen2.5-vl-3b-mjsynth-ocr-lora

This model is a LoRA fine-tuned version of Qwen/Qwen2.5-VL-3B-Instruct specifically optimized for Optical Character Recognition (OCR) tasks using the MJSynth dataset.

Model Details

Model Description

This model utilizes Parameter-Efficient Fine-Tuning (PEFT) via LoRA and 4-bit quantization to adapt the base Qwen2.5-VL-3B-Instruct model for highly accurate text recognition from images. It achieves significant improvements in Character Error Rate (CER) and Exact Match accuracy compared to the base model.

  • Developed by: phucdu
  • Model type: Vision-Language Model (VLM) / LoRA Adapter
  • Language(s) (NLP): English
  • License: Apache 2.0 (inheriting from Qwen)
  • Finetuned from model: Qwen/Qwen2.5-VL-3B-Instruct

Uses

Direct Use

Extracting and reading text from images (Optical Character Recognition - OCR). It is particularly tuned on single-word crops but generalizes well to standard text reading tasks.

How to Get Started with the Model

(Mục này đã được để trống theo yêu cầu của tác giả - Không chứa code tải mô hình).

Training Details

Training Data

A subset of priyank-m/MJSynth_text_recognition.

  • Pool size: 8000 examples
  • Split: 6400 Train / 800 Validation / 800 Test.

Training Procedure

Training Hyperparameters

  • Training regime: LoRA with 4-bit quantization (nf4, double quant) and bfloat16 compute dtype.
  • LoRA Rank (r): 16
  • LoRA Alpha: 32
  • LoRA Dropout: 0.05
  • Target Modules: down_proj, gate_proj, k_proj, o_proj, q_proj, up_proj, v_proj
  • Epochs: 3
  • Batch Size: 16 (Train) / 8 (Eval)
  • Learning Rate: 1e-4
  • LR Scheduler: Cosine
  • Warmup Ratio: 0.03
  • Weight Decay: 0.0

Speeds, Sizes, Times

  • Hardware: 1x NVIDIA L4 (24GB) GPU on Google Colab Pro.
  • Training Time: ~35 minutes (2115 seconds for 3 epochs).

Evaluation

Testing Data, Factors & Metrics

  • Testing Data: 800 held-out samples from the MJSynth dataset.
  • Metrics: Character Error Rate (CER), Word Error Rate (WER), Exact Match Accuracy.

Results

Comparing Base Model vs. Fine-tuned Model on the test set:

  • CER: 1.3056 ➔ 0.0842 (93.5% relative improvement)
  • WER: 2.3462 ➔ 0.1437 (93.8% relative improvement)
  • Exact Match Accuracy: 62.37% ➔ 85.62% (37.2% relative improvement)
  • Avg Inference Latency: ~0.768 sec/image

Summary

The LoRA fine-tuning process significantly enhanced the model's OCR capabilities on the target dataset, bringing the exact match accuracy up to over 85% while massively dropping both CER and WER.

Downloads last month
5
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for phucdu/qwen2.5-vl-3b-mjsynth-ocr-lora

Adapter
(339)
this model

Dataset used to train phucdu/qwen2.5-vl-3b-mjsynth-ocr-lora