Hugging Face's logo

meetween
/
SpeechLMM-v2.0-M-3B

Any-to-Any
Transformers
Safetensors
speechlmm
feature-extraction
multimodal
audio
speech
video
speech-recognition
speech-translation
lip-reading
meetween
custom_code
Model card Files
xet
 Community

Model Card for SpeechLMM-v2.0-M-3B

SpeechLMM-v2.0-M-3B is a 3B instruction-finetuned multimodal language model from the Meetween SpeechLMM 2.0 family, fine-tuned from Qwen2.5-Omni-3B for speech-centric virtual meetings. The SpeechLMM 2.0 family supports text, speech and video as input, and text and speech as output. The SpeechLMM 2.0 family is fine-tuned on a multitude of perception, understanding and generation tasks, including: automatic speech recognition, speech and machine translation, spoken and text question answering, speech and text summarisation, spoken language understanding, audio chaptering, audio-visual speaker diarisation, and lip-reading, covering ten EU languages (Czech, Dutch, English, French, German, Hungarian, Italian, Portuguese, Spanish, Swedish).

Model Details

Model Description

SpeechLMM-v2.0-M-3B is the 3B member of SpeechLMM 2.0, Meetween's second-generation multimodal foundation model for speech-centric virtual meetings. It is an auto-regressive instruction-finetuned model built on Qwen2.5-Omni-3B, extended with a dedicated AutoAVSR visual-speech (lip-reading) pathway and trainable adapters that project audio, video, and lip-reading video into the Thinker representation space.

SpeechLMM 2.0 models take multilingual text, speech, and video as input and can produce text or speech. They are fine-tuned for meeting-oriented perception, understanding and generation tasks, including automatic speech recognition, speech and machine translation, spoken and text question answering, speech and text summarisation, spoken language understanding (intent), audio chaptering, audio-visual speaker diarisation, lip-reading, text-to-speech, and speech-to-speech translation. Ten EU languages are officially supported: Czech, Dutch, English, French, German, Hungarian, Italian, Portuguese, Spanish, and Swedish.

This checkpoint follows a multi-stage training recipe: a lip-reading adapter stage, an audio-adapter stage, then multitask LoRA on the audio and video pathways as well as the language model.

  • Developed by: Nicola Bertoldi, Lorenzo Pratesi, Thomas Passera, Marco Comunità (Translated); Giuseppe Tanzi, Mubashir Shah (Pi School); Francesco Giuliari (Fondazione Bruno Kessler)
  • Funded by: European Commission (Horizon Europe, grant agreement No. 101135798, call HORIZON-CL4-2023-HUMAN-01-CNECT)
  • Shared by: Meetween consortium
  • Model type: Instruction-finetuned multimodal large language model (text, speech, and video)
  • Language(s): Czech, Dutch, English, French, German, Hungarian, Italian, Portuguese, Spanish, Swedish
  • License: CC-BY-NC-SA 4.0
  • Finetuned from model: Qwen/Qwen2.5-Omni-3B

Model Family

Model Backbone Params Input modalities Output modalities
SpeechLMM-v2.0-XL-30B Qwen3-Omni-30B-A3B-Instruct 30B Multilingual text and audio, English video Multilingual text and speech
SpeechLMM-v2.0-L-7B Qwen2.5-Omni-7B 7B Multilingual text and audio, English video Multilingual text and speech
SpeechLMM-v2.0-M-3B Qwen2.5-Omni-3B 3B Multilingual text and audio, English video Multilingual text and speech
SpeechLMM-v2.0-S-1.9B distilled from SpeechLMM-v2.0-M-3B 1.9B Multilingual text and audio, English video Multilingual text and speech

Model Sources

Uses

Direct Use

The model can be used as released, without further fine-tuning, as an instruction-finetuned multimodal LLM for speech-centric meeting and speech–language tasks. Intended direct users are researchers and developers who prompt the model with speech, text, and/or video and read back text (or speech) for a single task at a time.

Typical direct use cases are listed below, one per task. Ten EU languages are officially supported — Czech, Dutch, English, French, German, Hungarian, Italian, Portuguese, Spanish and Swedish. Tasks that cover a narrower set are annotated; the exact per-task coverage is given in Training Data.

  • ACHAP — Audio Chaptering: segment long audio into chapters, returning a start time and a title for each (English only).
  • ASR — Automatic Speech Recognition: transcribe speech into text in the same language.
  • AVSPEAKD — Audiovisual Speaker Diarisation: attribute speech to speakers using audio and video together (English only).
  • LIPREAD — Lip Reading / Visual Speech Recognition: transcribe speech from lip-crop video alone, without audio (English only).
  • MT — Machine Translation: translate text into another supported language.
  • S2ST — Speech-to-Speech Translation: translate spoken input into spoken output in another language.
  • SLU-I — Spoken Language Understanding (Intent): classify the speaker's intent from speech, against a dataset-defined intent list (English, German and French in training; evaluated also on Spanish, Hungarian, Dutch and Portuguese).
  • SQA-A — Spoken Question Answering, abstractive: answer a question about spoken content in free form (English input; English, German and Italian answers).
  • SQA-E — Spoken Question Answering, extractive: answer a question about spoken content by extracting the exact words from the input.
  • SQA-M — Spoken Question Answering, multiple choice: pick one answer option for a question about spoken content (English only).
  • SSUM — Speech Summarisation: summarise spoken input, such as a recorded meeting (English only).
  • ST — Speech Translation: translate spoken input into text in another language.
  • TQA-A — Textual Question Answering, abstractive: answer a question about written content in free form.
  • TQA-E — Textual Question Answering, extractive: answer a question about written content by extracting the exact words from the input.
  • TQA-M — Textual Question Answering, multiple choice: pick one answer option for a question about written content.
  • TSUM — Text Summarisation: summarise written input, such as a meeting transcript, into minutes or a summary (Czech and English only).
  • TTS — Text-to-Speech: generate speech from written input (English only).

Inference instructions can be found in the SpeechLMM inference repository.

Downstream Use

The intended downstream use is as a building block in a larger meeting-assistance system — such as Meetween's AI mediator — rather than as a stand-alone chat app.

Researchers and product teams may also further fine-tune (for example with LoRA) on in-domain meetings, additional languages, or a subset of tasks, using the SpeechLMM v2 training repository, and serve the result through the inference stack. Downstream deployers remain responsible for access control, logging, and how outputs are shown to meeting participants.

Out-of-Scope Use

The licence (CC-BY-NC-SA 4.0) excludes commercial use. Beyond that, the model should not be used, without human review, for:

  • decisions about people based on transcripts, summaries or minutes it produces; generated answers and summaries may contain spans that are not grounded in the input;
  • transcription of languages outside the ten officially supported ones, of heavily code-switched speech, or of accents that the training mix covers thinly;
  • lip-reading in production: visual-speech word error rate remains far above the usable range on all three evaluated sets (LipCrops 93.62, LRS2 54.58, VoxMM 100.17); see Lipread results.

People whose meetings, recordings, or video are passed to the model are affected users: it should not be run on personal data without an appropriate legal basis and consent.

How to Get Started with the Model

See Model Sources for the training, inference and evaluation repositories, and Prompts for the task prompts used at inference.

Training Details

Training Data

TASK Dataset Language License
ACHAP AMI Meeting Corpus en → en CC-BY-4.0
YTSeg en → en CC-BY-NC-SA-4.0
ASR CoVoST 2 cs → cs, de → de, en → en, es → es, fr → fr, hu → hu, it → it, nl → nl, pt → pt, sv → sv CC0-1.0
FLEURS cs → cs, de → de, en → en, es → es, fr → fr, hu → hu, it → it, nl → nl, pt → pt, sv → sv CC-BY-4.0
Multilingual Spoken Topical-Chat cs → cs, de → de, en → en, es → es, fr → fr, hu → hu, it → it, nl → nl, pt → pt, sv → sv CDLA-Sharing-1.0
Spoken DGT-TM cs → cs, de → de, en → en, es → es, fr → fr, hu → hu, it → it, nl → nl, pt → pt, sv → sv EUPL
VoxPopuli cs → cs, de → de, en → en, es → es, fr → fr, hu → hu, it → it, nl → nl CC0-1.0
EuroSpeech pt → pt, sv → sv Mixed Non-commercial
YTSeg en → en CC-BY-NC-SA-4.0
AVSPEAKD AMI Meeting Corpus en → en CC-BY-4.0
LIPREAD LipCrops en → en -
MT Europarl-ST {de, en, es, fr, it, nl, pt} → {de, en, es, fr, it, nl, pt} CC-BY-NC-4.0
SLU-I SLURP – Real & Synth en → en CC-BY-NC-4.0
Speech-MASSIVE de → de, fr → fr CC-BY-NC-SA-4.0
SQA-A LibriSQA – Part I – for IWSLT26 en → en, en → de, en → it -
SQA-E Multilingual Spoken SQuAD cs → cs, de → de, en → en, es → es, fr → fr, hu → hu, it → it, nl → nl, pt → pt, sv → sv CC-BY-SA-4.0
SSUM ICSI en → en CC-BY-4.0
ST Europarl-ST {de, en, es, fr, it, nl, pt} → {de, en, es, fr, it, nl, pt} CC-BY-NC-4.0
Multilingual Spoken Topical-Chat en → {cs, de, es, fr, hu, it, nl, pt, sv}, {cs, de, es, fr, hu, it, nl, pt, sv} → en CDLA-Sharing-1.0
Spoken DGT-TM en → {cs, de, es, fr, hu, it, nl, pt, sv}, {cs, de, es, fr, hu, it, nl, pt, sv} → en EUPL
TQA-M Multilingual Text LibriSQA – Part II cs → cs, de → de, en → en, es → es, fr → fr, hu → hu, it → it, nl → nl, pt → pt, sv → sv -
TSUM ICSI en → en CC-BY-4.0

There is no dedicated training data for TTS and S2ST. SpeechLMM-v2.0-M-3B reuses the Talker of Qwen2.5-Omni-3B unchanged: it stays frozen in every training stage and is never fine-tuned, so speech output comes entirely from the backbone.

Training Procedure

Training is organised in three stages. Each stage starts from the previous checkpoint. The Talker is frozen throughout and is never fine-tuned.

Stage 0A — Lip-reading adapter. Only the lip-reading adapter is trained, on English lip-crop video, with the lip-reading encoder frozen. This aligns visual speech features with the language model.

Stage 0B — Audio adapter. The audio adapter is fully fine-tuned on all audio tasks and languages. The audio encoder and language model stay frozen. This aligns audio speech features with the language model for the relevant domains and languages.

Stage 1 — Multitask LoRA. The main run continues full fine-tuning of the audio adapter and applies LoRA to the audio encoder and language model. Training covers all text, audio and video tasks, including audio-visual speaker diarisation. Lip-reading data is left out; the video path, lip-reading path, and Talker stay frozen.

Evaluation

Testing Data, Factors & Metrics

For evaluation across all tasks, languages, and datasets, small datasets are used so that the comparison stays manageable. Each small set contains 250 samples per language direction. These subsets match the full sets on some representative tasks (ASR, MT, ST): the mean is identical, and the difference in variance is negligible. The only evaluation set that is not a small subset is ICSI, which contains 6 samples.

Metrics are the standard ones for each benchmark: word and character error rate (WER, CER), BLEU, BLEURT, COMET, intent accuracy, exact match and token F1, and ROUGE. BLEURT and COMET are reported on a 0–100 scale. For S2ST and TTS, metrics prefixed with ASR- are computed on a Whisper large-v3 transcript of the generated speech.

Results below compare SpeechLMM-v2.0-M-3B with its Qwen2.5-Omni-3B backbone.

Results

ACHAP — Audio Chaptering

ACHAP is scored only on output that follows a fixed format — "MM:SS - chapter_title", one line per chapter. Qwen2.5-Omni-3B is not fine-tuned for this task and did not produce that format for most ICSI samples, so no score could be computed.

ICSI Source Lang Target Lang Collar F1 ↑ GC-BS ↑ TM-BS ↑ tm_matched ↑
Qwen2.5-Omni-3B en en - - - -
SpeechLMM-v2.0-M-3B en en 0.00 76.99 85.54 6.87

ASR — Automatic Speech Recognition

ACL6060 Source Language Target Language WER ↓ CER ↓
Qwen2.5-Omni-3B en en 15.98 9.47
SpeechLMM-v2.0-M-3B en en 13.64 7.48
CoVoST 2 Source Language Target Language WER ↓ CER ↓
Qwen2.5-Omni-3B de de 7.39 2.87
SpeechLMM-v2.0-M-3B de de 7.02 2.70
Qwen2.5-Omni-3B en en 9.30 5.35
SpeechLMM-v2.0-M-3B en en 9.30 4.78
Qwen2.5-Omni-3B es es 6.16 3.05
SpeechLMM-v2.0-M-3B es es 5.26 2.13
Qwen2.5-Omni-3B fr fr 9.05 3.15
SpeechLMM-v2.0-M-3B fr fr 8.25 2.83
Qwen2.5-Omni-3B it it 7.20 2.64
SpeechLMM-v2.0-M-3B it it 5.33 1.73
DiPCo Source Language Target Language WER ↓ CER ↓
Qwen2.5-Omni-3B en en 23.43 19.20
SpeechLMM-v2.0-M-3B en en 23.99 28.91
FLEURS Source Language Target Language WER ↓ CER ↓
Qwen2.5-Omni-3B cs cs 89.76 55.29
SpeechLMM-v2.0-M-3B cs cs 25.68 11.55
Qwen2.5-Omni-3B de de 8.52 4.67
SpeechLMM-v2.0-M-3B de de 5.83 2.28
Qwen2.5-Omni-3B en en 6.23 3.66
SpeechLMM-v2.0-M-3B en en 4.36 1.83
Qwen2.5-Omni-3B es es 5.78 2.82
SpeechLMM-v2.0-M-3B es es 3.77 1.50
Qwen2.5-Omni-3B fr fr 10.99 4.96
SpeechLMM-v2.0-M-3B fr fr 9.13 3.30
Qwen2.5-Omni-3B hu hu 274.05 240.05
SpeechLMM-v2.0-M-3B hu hu 34.72 16.24
Qwen2.5-Omni-3B it it 7.16 3.78
SpeechLMM-v2.0-M-3B it it 4.01 1.59
Qwen2.5-Omni-3B nl nl 13.88 7.59
SpeechLMM-v2.0-M-3B nl nl 7.11 2.90
Qwen2.5-Omni-3B pt pt 9.37 5.21
SpeechLMM-v2.0-M-3B pt pt 4.07 1.60
Qwen2.5-Omni-3B sv sv 102.05 64.32
SpeechLMM-v2.0-M-3B sv sv 19.74 7.82
Multilingual LibriSpeech Source Language Target Language WER ↓ CER ↓
Qwen2.5-Omni-3B de de 10.05 4.96
SpeechLMM-v2.0-M-3B de de 11.42 5.47
Qwen2.5-Omni-3B en en 8.37 5.06
SpeechLMM-v2.0-M-3B en en 9.12 4.75
Qwen2.5-Omni-3B es es 6.49 3.36
SpeechLMM-v2.0-M-3B es es 10.00 5.46
Qwen2.5-Omni-3B fr fr 8.14 4.31
SpeechLMM-v2.0-M-3B fr fr 8.76 4.33
Qwen2.5-Omni-3B it it 14.22 3.75
SpeechLMM-v2.0-M-3B it it 18.98 7.97
Qwen2.5-Omni-3B nl nl 15.13 6.19
SpeechLMM-v2.0-M-3B nl nl 13.19 5.14
Qwen2.5-Omni-3B pt pt 13.31 5.97
SpeechLMM-v2.0-M-3B pt pt 18.44 12.16

AVSPEAKD — Audio-Visual Speaker Diarisation

AVSPEAKD is scored only on output that follows a fixed format — "speaker_id: start_time - end_time", one line per speaker. Qwen2.5-Omni-7B is not fine-tuned for this task and did not produce that format for most VoxMM samples, so no score could be computed.

VoxMM Source Language Target Language F1 ↑ mAP ↑
Qwen2.5-Omni-3B en en - -
SpeechLMM-v2.0-M-3B en en 9.09 4.58

LIPREAD — Lip Reading / Visual Speech Recognition

Lip-reading is not a capability of the backbone Qwen2.5-Omni-3B. Visual speech recognition in SpeechLMM 2.0 comes from a dedicated AutoAVSR encoder and adapter added on top of Qwen-Omni; the backbone’s own vision encoder handles general video, not lip crops. Qwen2.5-Omni-3B therefore has no lip-reading pathway and was not evaluated on these sets.

LipCrops Source Language Target Language WER ↓ CER ↓
Qwen2.5-Omni-3B en en - -
SpeechLMM-v2.0-M-3B en en 93.62 69.74
LRS2 Source Language Target Language WER ↓ CER ↓
Qwen2.5-Omni-3B en en - -
SpeechLMM-v2.0-M-3B en en 54.58 36.59
VoxMM Source Language Target Language WER ↓ CER ↓
Qwen2.5-Omni-3B en en - -
SpeechLMM-v2.0-M-3B en en 100.17 74.49

MT — Machine Translation

ACL6060 Source Language Target Language BLEU ↑ BLEURT ↑ COMET ↑
Qwen2.5-Omni-3B en de 33.22 70.60 81.20
SpeechLMM-v2.0-M-3B en de 29.99 71.06 81.29
Qwen2.5-Omni-3B en fr 46.72 72.00 83.28
SpeechLMM-v2.0-M-3B en fr 36.80 68.97 81.81
Qwen2.5-Omni-3B en nl 36.77 74.01 83.96
SpeechLMM-v2.0-M-3B en nl 28.31 72.25 82.50
Qwen2.5-Omni-3B en pt 49.63 75.22 86.69
SpeechLMM-v2.0-M-3B en pt 37.83 72.80 84.75
FLORES+ Source Language Target Language BLEU ↑ BLEURT ↑ COMET ↑
Qwen2.5-Omni-3B en cs 17.37 66.57 81.10
SpeechLMM-v2.0-M-3B en cs 17.27 67.02 80.70
Qwen2.5-Omni-3B en de 30.36 74.26 85.53
SpeechLMM-v2.0-M-3B en de 31.24 74.54 85.38
Qwen2.5-Omni-3B en es 23.42 72.46 84.54
SpeechLMM-v2.0-M-3B en es 23.68 72.16 84.51
Qwen2.5-Omni-3B en fr 40.40 72.59 85.63
SpeechLMM-v2.0-M-3B en fr 38.18 72.44 86.02
Qwen2.5-Omni-3B en hu 6.94 60.87 64.90
SpeechLMM-v2.0-M-3B en hu 5.18 62.33 65.13
Qwen2.5-Omni-3B en it 23.36 72.91 84.89
SpeechLMM-v2.0-M-3B en it 23.34 74.00 85.24
Qwen2.5-Omni-3B en nl 20.20 70.50 83.73
SpeechLMM-v2.0-M-3B en nl 21.24 70.63 84.00
Qwen2.5-Omni-3B en pt 41.10 75.22 88.16
SpeechLMM-v2.0-M-3B en pt 33.78 74.46 87.66
Qwen2.5-Omni-3B en sv 24.07 65.74 79.40
SpeechLMM-v2.0-M-3B en sv 22.77 69.00 80.65
Qwen2.5-Omni-3B cs en 35.41 76.45 86.87
SpeechLMM-v2.0-M-3B cs en 32.32 75.40 86.33
Qwen2.5-Omni-3B de en 42.02 79.16 89.03
SpeechLMM-v2.0-M-3B de en 37.73 77.46 88.08
Qwen2.5-Omni-3B es en 29.51 76.49 86.84
SpeechLMM-v2.0-M-3B es en 27.25 75.95 86.44
Qwen2.5-Omni-3B fr en 40.86 79.63 89.19
SpeechLMM-v2.0-M-3B fr en 36.16 78.56 88.26
Qwen2.5-Omni-3B hu en 23.82 69.99 83.22
SpeechLMM-v2.0-M-3B hu en 20.45 68.68 81.98
Qwen2.5-Omni-3B it en 30.26 76.44 87.00
SpeechLMM-v2.0-M-3B it en 28.79 75.97 86.64
Qwen2.5-Omni-3B nl en 29.42 75.78 86.88
SpeechLMM-v2.0-M-3B nl en 27.15 75.14 86.43
Qwen2.5-Omni-3B pt en 44.86 79.68 88.96
SpeechLMM-v2.0-M-3B pt en 40.00 78.02 87.94
Qwen2.5-Omni-3B sv en 41.79 76.54 87.90
SpeechLMM-v2.0-M-3B sv en 37.57 75.47 86.99

SLU-I — Spoken Language Understanding (Intent)

SLURP Source Language Target Language Intent accuracy ↑
Qwen2.5-Omni-3B en en 41.6
SpeechLMM-v2.0-M-3B en en 81.2
Speech-MASSIVE Source Language Target Language Intent accuracy ↑
Qwen2.5-Omni-3B de de 43.6
SpeechLMM-v2.0-M-3B de de 80.0
Qwen2.5-Omni-3B es es 49.6
SpeechLMM-v2.0-M-3B es es 79.2
Qwen2.5-Omni-3B fr fr 51.2
SpeechLMM-v2.0-M-3B fr fr 82.8
Qwen2.5-Omni-3B hu hu 6.8
SpeechLMM-v2.0-M-3B hu hu 49.6
Qwen2.5-Omni-3B nl nl 44.4
SpeechLMM-v2.0-M-3B nl nl 83.6
Qwen2.5-Omni-3B pt pt 40.4
SpeechLMM-v2.0-M-3B pt pt 76.0

SQA-A — Spoken Question Answering (Abstractive)

LibriSQA – part I Source Language Target Language BertScore ↑
Qwen2.5-Omni-3B en en 55.82
SpeechLMM-v2.0-M-3B en en 68.47

SQA-E — Spoken Question Answering (Extractive)

SpokenSQuAD Source Language Target Language Exact Match ↑ Token F1 ↑
Qwen2.5-Omni-3B en en 70.4 83.19
SpeechLMM-v2.0-M-3B en en 76.8 85.00

SQA-M — Spoken Question Answering (Multiple Choice)

MMSU Source Language Target Language Accuracy ↑
Qwen2.5-Omni-3B en en 60.7
SpeechLMM-v2.0-M-3B en en 55.6

SSUM — Speech Summarisation

ICSI Source Language Target Language R-1 F1 ↑ R-2 F1 ↑ R-L F1 ↑
Qwen2.5-Omni-3B en en 8.78 1.58 7.90
SpeechLMM-v2.0-M-3B en en 10.09 2.37 9.87

ST — Speech Translation

ACL6060 Source Language Target Language BLEU ↑ BLEURT ↑ COMET ↑
Qwen2.5-Omni-3B en de 27.84 63.84 76.84
SpeechLMM-v2.0-M-3B en de 26.33 66.14 77.58
Qwen2.5-Omni-3B en fr 32.03 60.31 76.35
SpeechLMM-v2.0-M-3B en fr 28.13 61.82 77.67
Qwen2.5-Omni-3B en nl 23.22 62.80 76.86
SpeechLMM-v2.0-M-3B en nl 23.48 66.20 78.80
Qwen2.5-Omni-3B en pt 36.84 63.51 80.12
SpeechLMM-v2.0-M-3B en pt 34.27 64.81 80.98
CoVoST 2 Source Language Target Language BLEU ↑ BLEURT ↑ COMET ↑
Qwen2.5-Omni-3B en de 29.27 69.12 82.13
SpeechLMM-v2.0-M-3B en de 29.43 69.82 81.76
Qwen2.5-Omni-3B de en 39.69 74.11 85.49
SpeechLMM-v2.0-M-3B de en 36.20 73.09 84.64
Qwen2.5-Omni-3B es en 42.67 74.24 85.31
SpeechLMM-v2.0-M-3B es en 41.73 74.62 85.83
Qwen2.5-Omni-3B fr en 40.57 72.76 83.52
SpeechLMM-v2.0-M-3B fr en 39.91 73.23 84.18
Qwen2.5-Omni-3B it en 36.77 72.88 84.52
SpeechLMM-v2.0-M-3B it en 33.90 72.16 84.47
CVSS Source Language Target Language BLEU ↑ BLEURT ↑ COMET ↑
Qwen2.5-Omni-3B de en 23.31 69.94 81.36
SpeechLMM-v2.0-M-3B de en 21.33 69.33 81.20
Qwen2.5-Omni-3B es en 25.38 73.43 84.12
SpeechLMM-v2.0-M-3B es en 24.98 73.62 84.38
Qwen2.5-Omni-3B fr en 23.73 69.68 80.84
SpeechLMM-v2.0-M-3B fr en 22.10 69.88 80.80
Qwen2.5-Omni-3B it en 25.22 71.77 83.22
SpeechLMM-v2.0-M-3B it en 24.13 71.13 82.76
Qwen2.5-Omni-3B nl en 26.66 68.46 81.96
SpeechLMM-v2.0-M-3B nl en 25.31 69.79 82.75
Qwen2.5-Omni-3B pt en 36.33 75.56 86.21
SpeechLMM-v2.0-M-3B pt en 34.20 75.64 86.25
Qwen2.5-Omni-3B sv en 4.76 37.36 56.38
SpeechLMM-v2.0-M-3B sv en 16.92 60.79 74.78
Europarl-ST Source Language Target Language BLEU ↑ BLEURT ↑ COMET ↑
Qwen2.5-Omni-3B en de 20.55 62.00 80.19
SpeechLMM-v2.0-M-3B en de 19.44 62.51 81.41
Qwen2.5-Omni-3B en es 29.61 41.97 81.10
SpeechLMM-v2.0-M-3B en es 23.98 41.28 79.04
Qwen2.5-Omni-3B en fr 24.95 43.63 80.84
SpeechLMM-v2.0-M-3B en fr 29.34 46.53 85.53
Qwen2.5-Omni-3B en it 20.56 43.95 81.17
SpeechLMM-v2.0-M-3B en it 5.96 41.06 73.92
Qwen2.5-Omni-3B en nl 13.44 56.20 74.66
SpeechLMM-v2.0-M-3B en nl 17.48 62.04 80.98
Qwen2.5-Omni-3B en pt 22.10 25.26 77.63
SpeechLMM-v2.0-M-3B en pt 22.80 28.23 81.32
Qwen2.5-Omni-3B de en 29.66 58.63 82.56
SpeechLMM-v2.0-M-3B de en 22.60 56.44 79.94
Qwen2.5-Omni-3B es en 32.96 59.66 80.17
SpeechLMM-v2.0-M-3B es en 31.73 60.87 80.65
Qwen2.5-Omni-3B fr en 36.85 62.20 84.33
SpeechLMM-v2.0-M-3B fr en 34.56 60.89 82.66
Qwen2.5-Omni-3B it en 32.40 62.17 80.77
SpeechLMM-v2.0-M-3B it en 28.89 60.37 78.88
Qwen2.5-Omni-3B nl en 28.81 57.65 80.59
SpeechLMM-v2.0-M-3B nl en 26.50 57.02 80.95
Qwen2.5-Omni-3B pt en 32.59 61.34 80.93
SpeechLMM-v2.0-M-3B pt en 30.78 60.39 80.76
FLEURS+ Source Language Target Language BLEU ↑ BLEURT ↑ COMET ↑
Qwen2.5-Omni-3B en cs 11.65 58.93 74.48
SpeechLMM-v2.0-M-3B en cs 12.62 63.06 77.87
Qwen2.5-Omni-3B en de 25.53 71.87 83.71
SpeechLMM-v2.0-M-3B en de 26.42 72.41 84.40
Qwen2.5-Omni-3B en es 21.46 69.61 82.24
SpeechLMM-v2.0-M-3B en es 22.24 70.80 84.08
Qwen2.5-Omni-3B en fr 33.50 67.63 82.66
SpeechLMM-v2.0-M-3B en fr 33.90 68.88 84.21
Qwen2.5-Omni-3B en hu 4.60 60.55 64.38
SpeechLMM-v2.0-M-3B en hu 4.45 64.08 65.40
Qwen2.5-Omni-3B en it 21.17 70.29 82.93
SpeechLMM-v2.0-M-3B en it 21.56 72.26 84.47
Qwen2.5-Omni-3B en nl 17.12 66.48 80.50
SpeechLMM-v2.0-M-3B en nl 19.89 69.92 82.78
Qwen2.5-Omni-3B en pt 32.46 71.24 85.23
SpeechLMM-v2.0-M-3B en pt 31.04 72.18 86.09
Qwen2.5-Omni-3B en sv 21.49 61.71 75.63
SpeechLMM-v2.0-M-3B en sv 25.80 65.59 78.52
Qwen2.5-Omni-3B cs en 5.77 46.03 61.25
SpeechLMM-v2.0-M-3B cs en 24.41 65.41 78.66
Qwen2.5-Omni-3B de en 36.79 76.25 86.46
SpeechLMM-v2.0-M-3B de en 36.78 77.07 87.28
Qwen2.5-Omni-3B es en 24.85 74.44 84.91
SpeechLMM-v2.0-M-3B es en 26.59 75.55 86.01
Qwen2.5-Omni-3B fr en 32.76 75.83 86.51
SpeechLMM-v2.0-M-3B fr en 33.66 76.31 86.66
Qwen2.5-Omni-3B hu en 0.49 28.21 44.10
SpeechLMM-v2.0-M-3B hu en 13.25 54.46 70.41
Qwen2.5-Omni-3B it en 25.88 74.19 85.44
SpeechLMM-v2.0-M-3B it en 26.98 74.60 85.95
Qwen2.5-Omni-3B nl en 25.07 72.42 84.14
SpeechLMM-v2.0-M-3B nl en 25.13 73.60 85.24
Qwen2.5-Omni-3B pt en 40.02 75.87 86.83
SpeechLMM-v2.0-M-3B pt en 39.76 77.20 87.70
Qwen2.5-Omni-3B sv en 8.37 41.10 56.16
SpeechLMM-v2.0-M-3B sv en 29.22 69.04 82.40

S2ST — Speech-to-Speech Translation

ASR-BLEU, ASR-BLEURT and ASR-COMET are the ST metrics computed on a Whisper large-v3 transcript of the generated speech, scored against the text reference.

Note on speech output. SpeechLMM-v2.0-M-3B reuses the Talker of Qwen2.5-Omni-3B unchanged: it is frozen in every training stage and is never fine-tuned, so the speech scored below is produced by the backbone Talker. That Talker produces reliable speech in English and Chinese only. Hence, only directions into English are reported below.

CVSS Source Language Target Language ASR-BLEU ↑ ASR-BLEURT ↑ ASR-COMET ↑
Qwen2.5-Omni-3B de en 20.88 66.02 78.82
SpeechLMM-v2.0-M-3B de en 20.01 64.72 77.12
Qwen2.5-Omni-3B es en 25.99 71.47 82.17
SpeechLMM-v2.0-M-3B es en 23.91 71.31 82.00
Qwen2.5-Omni-3B fr en 19.90 65.73 75.79
SpeechLMM-v2.0-M-3B fr en 19.80 64.14 74.88
Qwen2.5-Omni-3B it en 23.57 69.09 80.93
SpeechLMM-v2.0-M-3B it en 21.65 68.40 80.58
Qwen2.5-Omni-3B nl en 21.06 67.61 80.06
SpeechLMM-v2.0-M-3B nl en 21.83 70.11 82.30
Qwen2.5-Omni-3B pt en 27.86 71.84 82.67
SpeechLMM-v2.0-M-3B pt en 30.73 72.58 83.13
Qwen2.5-Omni-3B sv en 2.41 35.78 52.94
SpeechLMM-v2.0-M-3B sv en 15.55 57.22 72.25
FLEURS+ Source Language Target Language ASR-BLEU ↑ ASR-BLEURT ↑ ASR-COMET ↑
Qwen2.5-Omni-3B cs en 7.33 40.72 57.24
SpeechLMM-v2.0-M-3B cs en 20.93 63.64 76.64
Qwen2.5-Omni-3B de en 34.10 73.57 84.34
SpeechLMM-v2.0-M-3B de en 33.15 73.48 84.29
Qwen2.5-Omni-3B es en 24.85 72.29 83.88
SpeechLMM-v2.0-M-3B es en 25.55 72.69 84.40
Qwen2.5-Omni-3B fr en 32.03 73.27 84.49
SpeechLMM-v2.0-M-3B fr en 31.96 73.63 84.53
Qwen2.5-Omni-3B hu en 0.33 24.71 41.67
SpeechLMM-v2.0-M-3B hu en 12.11 51.90 67.68
Qwen2.5-Omni-3B it en 26.01 73.16 84.57
SpeechLMM-v2.0-M-3B it en 27.12 73.66 84.65
Qwen2.5-Omni-3B nl en 24.18 69.99 82.70
SpeechLMM-v2.0-M-3B nl en 23.83 70.02 82.95
Qwen2.5-Omni-3B pt en 37.66 73.48 84.56
SpeechLMM-v2.0-M-3B pt en 37.56 74.07 85.51
Qwen2.5-Omni-3B sv en 6.52 37.22 52.07
SpeechLMM-v2.0-M-3B sv en 27.27 66.24 80.24

TSUM — Text Summarisation

ICSI Source Language Target Language R-1 F1 ↑ R-2 F1 ↑ R-L F1 ↑
Qwen2.5-Omni-3B en en 16.54 1.90 14.83
SpeechLMM-v2.0-M-3B en en 12.99 2.34 12.13

TTS — Text-to-Speech

ASR-WER and ASR-CER are computed using a Whisper large-v3 transcript of the generated speech against the input text.

Note on speech output. SpeechLMM-v2.0-M-3B reuses the Talker of Qwen2.5-Omni-3B unchanged: it is frozen in every training stage and is never fine-tuned, so the speech scored below is produced by the backbone Talker. That Talker produces reliable speech in English and Chinese only. Hence, only directions into English are reported below.

LJSpeech Source Language Target Language ASR-WER ↓ ASR-CER ↓ UTMOS ↑
Qwen2.5-Omni-3B en en 23.93 21.71 4.37
SpeechLMM-v2.0-M-3B en en 23.82 21.85 4.33

Summary

Against Qwen2.5-Omni-3B, fine-tuning brings the largest gains on speech recognition and spoken-language understanding. On FLEURS, word error rate falls in all ten languages, and the three languages where the backbone effectively fails are repaired: Hungarian 274.05 → 34.72, Swedish 102.05 → 19.74, Czech 89.76 → 25.68. Spoken-intent accuracy roughly doubles on SLURP (41.60 → 81.20) and rises on every evaluated Speech-MASSIVE language, by 30 to 43 points. Extractive spoken question answering on SpokenSQuAD improves (exact match 70.40 → 76.80), and meeting summarisation on ICSI improves on all three ROUGE measures, although absolute scores stay low. Speech translation on FLEURS+ gains on 14 of 18 directions in BLEU and on all 18 in BLEURT and COMET, again concentrated on the directions the backbone handles worst (sv→en +20.85, cs→en +18.64, hu→en +12.76).

The trade-offs run the other way on text-centric and high-resource benchmarks. Machine translation on FLORES+ loses BLEU on 15 of 18 directions (worst en→pt −7.32) and on all four ACL6060 directions; multiple-choice spoken question answering on MMSU drops 60.70 → 55.60; and recognition on Multilingual LibriSpeech degrades on six of seven languages. Text summarisation on ICSI is mixed, losing R-1 and R-L while gaining R-2. Speech-to-speech translation is close to the backbone on CVSS and ahead on FLEURS+ in BLEURT and COMET. Text-to-speech on LJSpeech is essentially unchanged, as expected from a frozen Talker. Lip-reading word error rate remains far above a usable range on all three visual-speech sets (LipCrops 93.62, LRS2 54.58, VoxMM 100.17).

Prompts

The prompts used for inference and training, per task.

Task Inference prompt example Training prompt template
ACHAP Given the English speech input, identify the chapters. Return one chapter per line in the format: MM:SS - chapter_title. Use HH:MM:SS for timestamps of one hour or longer. Each timestamp must indicate the chapter's start within the audio. Write chapter titles in English and return them in chronological order. Return only the timestamp and title lines, without a transcript, Markdown headings, or additional commentary. Given the {lang_in} {format_in} input, identify the chapters. For each chapter, return its start time and title in this format: 'start_time - chapter_title'. Chapter titles must be written in {lang_out}. Return the chapters in chronological order. Input: {input}
ASR Transcribe the German speech input. The transcription must be written in German. Return only the transcription. Transcribe the {lang_in} {format_in} input. The transcription must be written in {lang_in} Input: {input}
LIPREAD Transcribe the speech visible in the English video input. The transcription must be written in English. Transcribe the speech visible in the {lang_in} {format_in} input. The transcription must be written in {lang_in}. Input: {input}
MT Translate the Czech text input into English text. Preserve the meaning of the original input. Return only the translation. Translate the {lang_in} {format_in} input into {lang_out} {format_out}. Preserve the meaning of the original input. Input: {input}
SLU-I Given the German speech, identify the speaker's intent. Choose exactly one intent from this list: alarm_query, alarm_remove, alarm_set, audio_volume_down, … weather_query, wemo_off. If the intent is not present, reply with 'unanswerable'. Return only the exact intent label. (the intent list is defined by the dataset) Given the {lang_in} {format_in}, identify the speaker's intent. Choose exactly one intent from the following intent list. If the intent is not present in the list, reply with 'unanswerable'. Input: {input} Intent list: {intents}
SQA-A Given the English speech input, answer the accompanying question. Base the answer only on information contained in the input. Answer in English, concisely but with all requested details. Return only the answer. Given the {lang_in} {format_in} input, answer the {lang_qst} {format_qst} question. Base the answer only on the information contained in the input. The answer must be written in {lang_out}. Input: {input} Question: {question}
SQA-E Given the English speech input, answer the accompanying question. Extract the exact words from the input that answer the question. Do not paraphrase or add words. The answer must be written in English. If absent, reply with 'unanswerable'. Given the {lang_in} {format_in} input, answer the {lang_qst} {format_qst} question. Extract the exact words from the input that answer the question. Do not paraphrase, modify, or add any words. The answer must be written in {lang_out}. If the answer is not present in the input, reply with 'unanswerable'. Input: {input} Question: {question}
SQA-M Given the input, answer the text question. Choose exactly one answer from the following options. Return only the answer label: A, B, C, or D. (the actual answer choices are given within the input) Given the {lang_in} {format_in} input, answer the {lang_qst} {format_qst} question. Choose exactly one answer from the following options. Input: {input} Question: {question} Options: {options}
SSUM Summarize the following English speech input. Write the summary in English. Return only the summary. Summarize the {lang_in} {format_in} input. The summary must be written in {lang_out}. Input: {input}
ST Translate the German speech input into English text. Preserve the meaning of the original input. Return only the translation. Translate the {lang_in} {format_in} input into {lang_out} {format_out}. Preserve the meaning of the original input. Input: {input}
S2ST Translate the Czech speech input into English speech. Preserve the meaning of the original input. Return only the translation, and then stop. —
TSUM Summarize the following English text input. Write the summary in English. Return only the summary. Summarize the {lang_in} {format_in} input. The summary must be written in {lang_out}. Input: {input}
TTS Generate English speech from the English text input.\nSpeak only the text input, and then stop. —

Environmental Impact

Carbon emissions can be estimated using the Machine Learning Impact calculator presented in Lacoste et al. (2019).

Technical Specifications

Model Architecture and Objective

SpeechLMM-v2.0-M-3B uses Qwen2.5-Omni-3B as the multimodal backbone. Qwen2.5-Omni is a Thinker–Talker model that unifies perception, understanding and generation across text, image, audio, and video:

  • The Thinker handles multimodal reasoning and text generation (36 layers, hidden size 2048, 3.40B parameters).
  • The Talker produces streaming speech tokens conditioned on hidden states from the Thinker (24 layers, hidden size 896, embedding width 2048; used for TTS and speech-to-speech translation).
  • The audio encoder is initialised from Whisper-large-v3 (32 layers, width 1280) and processes audio in blocks, so long inputs can be encoded without rebuilding alignment to the language model. The audio and vision trunks are the same as in SpeechLMM-v2.0-L-7B; only the projection into the Thinker changes, to width 2048.
  • The vision encoder is the ViT from Qwen2.5-VL. In SpeechLMM it is used for general video (for example audio-visual speaker diarisation), not for lip-reading.
  • The lip-reading encoder is AutoAVSR, used for visual speech / lip crops.

The backbone already covers a wide set of written and spoken languages. SpeechLMM still trains for ten EU languages, including Czech, Hungarian, and Swedish. Updating the audio encoder and adapter (Stages 0B and 1) is how those languages are brought into the speech-understanding path.

On top of Qwen2.5-Omni, SpeechLMM v2 adds the pieces needed for meeting scenarios:

Component Role
Backbone Qwen2.5-Omni-3B (Thinker + Talker + Whisper-based audio encoder + Qwen2.5-VL vision encoder)
Lip-reading encoder AutoAVSR for visual speech / lip crops
Adapters Trainable projections from audio, video, and lip-read encoder outputs into Thinker space

The dedicated AutoAVSR pathway is the main architectural addition relative to Qwen2.5-Omni: it supplies visual speech features when audio is degraded or absent, separate from the Qwen2.5-VL vision tower used for general video.

Citation

Referenced works:

Glossary

Abbreviation Meaning
ACHAP Audio Chaptering
ASR Automatic Speech Recognition
AutoAVSR Automatic Audio-Visual Speech Recognition
AVSPEAKD Audiovisual Speaker Diarisation
BLEU Bilingual Evaluation Understudy
CER Character Error Rate
DST Downstream Tasks
LLM Large Language Model
LIPREAD Lip Reading
LoRA Low-Rank Adaptation
MT Machine Translation
PIER Embedded-language Error Rate (code-switching)
R-1 / R-2 / R-L ROUGE-1 / ROUGE-2 / ROUGE-L
S2ST Speech-to-Speech Translation
SLU-I Spoken Language Understanding – Intent
SQA Spoken Question Answering
SQA-A Spoken Question Answering – Abstractive
SQA-E Spoken Question Answering – Extractive
SQA-M Spoken Question Answering – Multiple Choice
SSUM Speech Summarization
ST Speech Translation
TQA Textual Question Answering
TQA-A Textual Question Answering – Abstractive
TQA-E Textual Question Answering – Extractive
TQA-M Textual Question Answering – Multiple Choice
TSUM Text Summarization
TTS Text-to-Speech
VSR Visual Speech Recognition
WER Word Error Rate
Downloads last month
31
Safetensors
Model size
6B params
Tensor type
BF16
·
Inference Providers NEW
Any-to-Any
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for meetween/SpeechLMM-v2.0-M-3B

Base model

Qwen/Qwen2.5-Omni-3B
Finetuned
(43)
this model

Papers for meetween/SpeechLMM-v2.0-M-3B