cstr/granite-speech-4.0-1b-GGUF

Granite-Speech-4.0-1B — GGUF (CrispASR)

GGUF conversions of ibm-granite/granite-speech-4.0-1b, IBM's 1B-parameter speech-LLM (Conformer encoder + BLIP-2 Q-Former projector + Granite μP LLM head). Runs through CrispASR's --backend granite path with full feature support including the new prompt-based speech-translation path.

What's in the box

File	Size	Quantization	Notes
granite-speech-4.0-1b-f16.gguf	5.58 GB	F16	Reference precision
granite-speech-4.0-1b-q8_0.gguf	3.86 GB	Q8_0	Effectively lossless
granite-speech-4.0-1b-q5_0.gguf	3.17 GB	Q5_0	Slightly slower than Q4_K but a bit more accurate
granite-speech-4.0-1b-q4_k.gguf	2.94 GB	Q4_K	1.9× compressed; smallest reasonable choice

All four contain:

• The Conformer audio encoder (16 layers, d_model 1024, Shaw-style relative-position attention with 200-sample block-local context)
• The BLIP-2 Q-Former projector (2 layers, 16 heads, 3-query learnable prompt tokens, audio→text dim conversion)
• The Granite-4.0 1B LLM (40 layers, d_model 2048, 16 heads / 4 KV heads, 8192 ff, 100353 vocab, μP scaling)
• The full GPT-2-style BPE vocab and merges table (the merges are new in this re-conversion — older GGUFs had vocab strings only)
• Mel filterbank (80 bins, stacked 2× to 160), Hann window

What's new in this conversion

• tokenizer.ggml.merges field — the previous granite GGUFs had only tokenizer.ggml.tokens. The C++ side could detokenize but couldn't tokenize arbitrary text, which blocked the prompt-based translate path. This re-conversion writes the merges table from merges.txt and unblocks --translate -tl <lang> end-to-end.

If you have an older granite GGUF without the merges field, the C++ side still loads it for plain transcribe, but --translate is silently a no-op. Re-download these files to get translate.

Use with CrispASR (no Python at runtime)

# Build crispasr (one-time)
git clone https://github.com/CrispStrobe/CrispASR
cd CrispASR
cmake -B build -DCMAKE_BUILD_TYPE=Release
cmake --build build -j$(nproc) --target whisper-cli

# Auto-download the recommended quant on first use:
./build/bin/crispasr --backend granite -m auto -f my_audio.wav

# Or point at a local file:
./build/bin/crispasr --backend granite \
    -m granite-speech-4.0-1b-q4_k.gguf \
    -f my_audio.wav

# Speech translation to German via the runtime-tokenized prompt path:
./build/bin/crispasr --backend granite \
    -m granite-speech-4.0-1b-q4_k.gguf \
    -f my_audio.wav --translate -tl de
# → "und so meine amerikaner, fragen sie nicht, was ihr land für sie
#    tun kann, fragen sie, was sie für ihr land tun können."

# Word-level timestamps via the canary CTC aligner second pass:
./build/bin/crispasr --backend granite \
    -m granite-speech-4.0-1b-q4_k.gguf \
    -f my_audio.wav -am canary-ctc-aligner-q5_0.gguf -osrt -ml 1

Pure C++ runtime: no Python, no PyTorch, no NeMo. The only build deps beyond C++17 + CMake are libcurl or wget for auto-download and (optionally) BLAS / OpenMP for the encoder hot-loop speedup.

Languages

Same as upstream Granite-Speech 4.0: English, French, German, Spanish, Portuguese, Japanese.

How it was made

# 1. Download the base model from HF
hf download ibm-granite/granite-speech-4.0-1b --local-dir ./granite-speech-4.0-1b

# 2. Convert to F16 GGUF (the new converter writes both vocab and merges)
python models/convert-granite-speech-to-gguf.py \
    --input ./granite-speech-4.0-1b \
    --output granite-speech-4.0-1b-f16.gguf

# 3. Quantize
./build/bin/crispasr-quantize granite-speech-4.0-1b-f16.gguf granite-speech-4.0-1b-q8_0.gguf q8_0
./build/bin/crispasr-quantize granite-speech-4.0-1b-f16.gguf granite-speech-4.0-1b-q5_0.gguf q5_0
./build/bin/crispasr-quantize granite-speech-4.0-1b-f16.gguf granite-speech-4.0-1b-q4_k.gguf q4_k

Total: 940 tensors per file (encoder, projector, LLM, audio frontend, BPE tables).

Verification

All four quants tested on samples/jfk.wav under CrispASR's granite backend:

default transcribe (any quant):
  "and so my fellow americans ask not what your country can do for you
   ask what you can do for your country"

translate -> German (any quant):
  "und so meine amerikaner, fragen sie nicht, was ihr land für sie
   tun kann, fragen sie, was sie für ihr land tun können."

Note: Granite-Speech is a μP-scaled model — the C++ implementation respects the embedding multiplier, attention multiplier, residual multiplier, and logits scale from the upstream config. Bit-identity is preserved across quants on the default transcribe path within q4_k expected float drift.

License

Apache-2.0, same as upstream IBM Granite-Speech-4.0-1B.

Citation

@misc{granite-speech,
    title  = {Granite-Speech-4.0-1B},
    author = {IBM Research},
    year   = {2026},
    url    = {https://huggingface.co/ibm-granite/granite-speech-4.0-1b}
}

Provenance and EU AI Act Art. 53 note

• Upstream model: ibm-granite/granite-speech-4.0-1b.
• Upstream licence: apache-2.0. This repository redistributes under the same terms; it grants no rights the upstream licence does not.
• What was done here: format conversion and/or quantisation only (GGUF/GGML). No training, no fine-tuning, no merging, no distillation, no change to architecture, vocabulary or capability. Only the numeric representation of the upstream weights differs.
• Training data: documented — where it is documented at all — by the upstream provider; see the upstream model card. No training data was used, added or selected by this repository.
• Provider status: under Regulation (EU) 2024/1689 the upstream authors remain the provider of this model. Converting the serialisation format does not make this repository the provider of a new general-purpose AI model, and no such claim is made. Questions about training content, copyright policy or model capability belong upstream.