SmolVLM2-2.2B — LiteRT-LM (on-device Vision-Language Model)

HuggingFaceTB/SmolVLM2-2.2B-Instruct (image path) converted to the LiteRT-LM (.litertlm) format for on-device image+text inference with Google's LiteRT-LM runtime.

SmolVLM2-2.2B is the largest / most capable of Hugging Face's SmolVLM2 family: a SigLIP vision encoder + pixel-shuffle connector feeding a SmolLM2-1.7B (Llama-architecture) language decoder. Give it an image and a question, get a grounded answer, fully offline.

File	SmolVLM2-2.2B.litertlm
Vision	SigLIP encoder (384×384, patch 14 → 729 patches, no CLS) + pixel-shuffle ×3 + Linear connector, int8 → 81 image tokens
Decoder	SmolLM2-1.7B (Llama, 2048-dim, 24 layers), int4 weights (blockwise-32 + OCTAV); tied embedding INT8 (externalized)
Compute	integer
Context (KV cache)	2048
Image input	resized to 384×384 ((x−0.5)/0.5 normalization baked into the vision encoder)
Base model	HuggingFaceTB/SmolVLM2-2.2B-Instruct

How to use

1. Install the runtime

pip install litert-lm

2. Run it in one command — this downloads the bundle, encodes your image and answers:

litert-lm run --from-huggingface-repo litert-community/SmolVLM2-2.2B SmolVLM2-2.2B.litertlm \
  --attachment photo.jpg \
  --prompt "Describe this image in one sentence."

On the COCO sample image huggingface/documentation-images/coco_sample.png (two tabby cats on a pink blanket, remote controls beside them) this prints:

Two cats are sleeping on a pink blanket with remote controls nearby.

Drop --prompt for an interactive chat, and pass --attachment more than once for several images. litert-lm serve exposes the same bundle as a local OpenAI-compatible API. The same file runs on macOS, Linux and Windows.

Performance

litert-lm benchmark (litert-lm 0.15.0) on an Apple M4 Max, -p 256 -d 256 --runs 3 (the tool averages three iterations), max-num-tokens 4096, warm-up run discarded, otherwise idle machine. These figures cover the text path; the vision encoder runs once per image and is not included.

Device	Backend	Prefill (256)	Decode	TTFT
Apple M4 Max (macOS)	CPU	119 tok/s	33.6 tok/s	2.34 s
Apple M4 Max (macOS)	GPU (Metal)	1608 tok/s	134.7 tok/s	0.18 s

Every desktop backend listed above was checked by actually generating a caption on it, not just by reading the benchmark tool's output.

Accuracy note

Single-image VQA produces coherent, image-grounded answers (the SigLIP vision tower converts bit-faithfully to the reference, float CPU-parity corr ≈ 1.0). This is the largest SmolVLM2, so it is notably more capable than SmolVLM2-500M.

⚠️ Best for single-image VQA — one image per conversation

Ask about one image per chat (start a new conversation for a different image).

Run on Android — Google AI Edge Gallery

Update (July 2026): Google AI Edge Gallery v1.0.16+ can import litert-lm models directly from Hugging Face inside the app (tap +) — no computer or adb needed. The manual steps below are only required on older builds or for sideloading a local file.

Run this model with image input in the official Google AI Edge Gallery app — no custom app needed:

1. Push the bundle onto the phone (or download it there directly from this repo): adb push SmolVLM2-2.2B.litertlm /sdcard/Download/
2. Open the Gallery app, tap the + icon (bottom-right) and pick SmolVLM2-2.2B.litertlm.
3. In the Import Model dialog, check "Support image" (required for image input), then tap Import.
4. Open the Ask Image task, choose the imported model, attach a photo, and ask.

Run on desktop (LiteRT-LM CLI)

The same .litertlm bundle runs on macOS / Linux / Windows with the official LiteRT-LM CLI — including as a local OpenAI-compatible API server:

pip install litert-lm
litert-lm import --from-huggingface-repo litert-community/SmolVLM2-2.2B SmolVLM2-2.2B.litertlm smolvlm2-2.2b
litert-lm run smolvlm2-2.2b     # interactive chat in the terminal
litert-lm serve           # local OpenAI-compatible API server

Run on iPhone / macOS

Use the LiteRT-LM Swift runtime (swift-litert-lm / the LiteRTDemo sample). Load SmolVLM2-2.2B.litertlm with the vision tower enabled (modalities Modality.textImage / [.vision]), attach a photo, and ask.

Conversion notes

• LiteRT-LM fast_vlm bundle: VISION_ENCODER ([1,384,384,3]→SigLIP) + VISION_ADAPTER (pixel-shuffle ×3 + Linear → [1,81,2048]) + single-token EMBEDDER + PREFILL_DECODE.
• The vision encoder uses the static position-embedding path (the model's dynamic bucketize position logic is bypassed — numerically identical for a full 384×384 frame) and bakes the (x−0.5)/0.5 normalization + NCHW transpose into the graph.
• Single-image, no high-res splitting → a fixed 81 soft tokens; SmolLM2 (Llama) decoder exported with externalized (tied) embedder.

License

Apache-2.0 (SmolVLM2 + SmolLM2). See the base model card.