Model Card for SpeechLMM-v2.0-L-7B

SpeechLMM-v2.0-L-7B is a 7B instruction-finetuned multimodal language model from the Meetween SpeechLMM 2.0 family, fine-tuned from Qwen2.5-Omni-7B for speech-centric virtual meetings. The SpeechLMM 2.0 family supports text, speech and video as input, and text and speech as output. The SpeechLMM 2.0 family is fine-tuned on a multitude of perception, understanding and generation tasks, including: automatic speech recognition, speech and machine translation, spoken and text question answering, speech and text summarisation, spoken language understanding, audio chaptering, audio-visual speaker diarisation, and lip-reading, covering ten EU languages (Czech, Dutch, English, French, German, Hungarian, Italian, Portuguese, Spanish, Swedish).

Model Details

Model Description

SpeechLMM-v2.0-L-7B is the 7B member of SpeechLMM 2.0, Meetween's second-generation multimodal foundation model for speech-centric virtual meetings. It is an auto-regressive instruction-finetuned model built on Qwen2.5-Omni-7B, extended with a dedicated AutoAVSR visual-speech (lip-reading) pathway and trainable adapters that project audio, video, and lip-reading video into the Thinker representation space.

SpeechLMM 2.0 models take multilingual text, speech, and video as input and can produce text or speech. They are fine-tuned for meeting-oriented perception, understanding and generation tasks, including automatic speech recognition, speech and machine translation, spoken and text question answering, speech and text summarisation, spoken language understanding (intent), audio chaptering, audio-visual speaker diarisation, lip-reading, text-to-speech, and speech-to-speech translation. Ten EU languages are officially supported: Czech, Dutch, English, French, German, Hungarian, Italian, Portuguese, Spanish, and Swedish.

This checkpoint follows a multi-stage training recipe: a lip-reading adapter stage, an audio-adapter stage, then multitask LoRA on the audio and video pathways as well as the language model.

Developed by: Nicola Bertoldi, Lorenzo Pratesi, Thomas Passera, Marco Comunità (Translated); Giuseppe Tanzi, Mubashir Shah (Pi School); Francesco Giuliari (Fondazione Bruno Kessler)
Funded by: European Commission (Horizon Europe, grant agreement No. 101135798, call HORIZON-CL4-2023-HUMAN-01-CNECT)
Shared by: Meetween consortium
Model type: Instruction-finetuned multimodal large language model (text, speech, and video)
Language(s): Czech, Dutch, English, French, German, Hungarian, Italian, Portuguese, Spanish, Swedish
License: CC-BY-NC-SA 4.0
Finetuned from model: Qwen/Qwen2.5-Omni-7B

Model Family

Model	Backbone	Params	Input modalities	Output modalities
SpeechLMM-v2.0-XL-30B	Qwen3-Omni-30B-A3B-Instruct	30B	Multilingual text and audio, English video	Multilingual text and speech
SpeechLMM-v2.0-L-7B	Qwen2.5-Omni-7B	7B	Multilingual text and audio, English video	Multilingual text and speech
SpeechLMM-v2.0-M-3B	Qwen2.5-Omni-3B	3B	Multilingual text and audio, English video	Multilingual text and speech
SpeechLMM-v2.0-S-1.9B	distilled from SpeechLMM-v2.0-M-3B	1.9B	Multilingual text and audio, English video	Multilingual text and speech

Model Sources

Repository:
Collection: https://huggingface.co/meetween/
Project: Meetween — My Personal AI Mediator for Virtual Meetings

Uses

Direct Use

The model can be used as released, without further fine-tuning, as an instruction-finetuned multimodal LLM for speech-centric meeting and speech–language tasks. Intended direct users are researchers and developers who prompt the model with speech, text, and/or video and read back text (or speech) for a single task at a time.

Typical direct use cases are listed below, one per task. Ten EU languages are officially supported — Czech, Dutch, English, French, German, Hungarian, Italian, Portuguese, Spanish and Swedish. Tasks that cover a narrower set are annotated; the exact per-task coverage is given in Training Data.

ACHAP — Audio Chaptering: segment long audio into chapters, returning a start time and a title for each (English only).
ASR — Automatic Speech Recognition: transcribe speech into text in the same language.
AVSPEAKD — Audiovisual Speaker Diarisation: attribute speech to speakers using audio and video together (English only).
LIPREAD — Lip Reading / Visual Speech Recognition: transcribe speech from lip-crop video alone, without audio (English only).
MT — Machine Translation: translate text into another supported language.
S2ST — Speech-to-Speech Translation: translate spoken input into spoken output in another language.
SLU-I — Spoken Language Understanding (Intent): classify the speaker's intent from speech, against a dataset-defined intent list (English, German and French in training; evaluated also on Spanish, Hungarian, Dutch and Portuguese).
SQA-A — Spoken Question Answering, abstractive: answer a question about spoken content in free form (English input; English, German and Italian answers).
SQA-E — Spoken Question Answering, extractive: answer a question about spoken content by extracting the exact words from the input.
SQA-M — Spoken Question Answering, multiple choice: pick one answer option for a question about spoken content (English only).
SSUM — Speech Summarisation: summarise spoken input, such as a recorded meeting (English only).
ST — Speech Translation: translate spoken input into text in another language.
TQA-A — Textual Question Answering, abstractive: answer a question about written content in free form.
TQA-E — Textual Question Answering, extractive: answer a question about written content by extracting the exact words from the input.
TQA-M — Textual Question Answering, multiple choice: pick one answer option for a question about written content.
TSUM — Text Summarisation: summarise written input, such as a meeting transcript, into minutes or a summary (Czech and English only).
TTS — Text-to-Speech: generate speech from written input (English only).

Inference instructions can be found in the SpeechLMM inference repository.

Downstream Use

The intended downstream use is as a building block in a larger meeting-assistance system — such as Meetween's AI mediator — rather than as a stand-alone chat app.

Researchers and product teams may also further fine-tune (for example with LoRA) on in-domain meetings, additional languages, or a subset of tasks, using the SpeechLMM v2 training repository, and serve the result through the inference stack. Downstream deployers remain responsible for access control, logging, and how outputs are shown to meeting participants.

Out-of-Scope Use

The licence (CC-BY-NC-SA 4.0) excludes commercial use. Beyond that, the measurements in Bias, Risks, and Limitations indicate that the model should not be used, without human review, for:

decisions about people based on transcripts, summaries or minutes it produces; the hallucination probes show that generated answers and summaries contain ungrounded spans at a rate comparable to the backbone; see Hallucination and Groundedness;
transcription of languages outside the ten officially supported ones, of heavily code-switched speech, or of accents shown to be poorly covered; see Code Switching and Accented Speech Recognition;
lip-reading in production: visual-speech word error rate remains far above the usable range on all evaluated sets; see Lipread results.

People whose meetings, recordings, or video are passed to the model are affected users: it should not be run on personal data without an appropriate legal basis and consent.

Bias, Risks, and Limitations

This section reports the bias assessment and criticality scenarios. Three families of checks were run: gender bias in ASR, hallucination/groundedness in question answering and summarisation, and language/data-distribution bias (code-switching and accented speech). Comparisons use Qwen2.5-Omni-7B throughout, and Phi-4 and SpeechLMM-v1.0-L where those runs exist.

Gender Bias in ASR

A key potential vulnerability of language technologies is gender bias, which can lead to unequal performance and treatment across gender expressions and groups. This issue is also notable in speech-related tasks such as ASR, where models may exhibit performance differences based on vocal pitch — sometimes significantly preferring either lower-pitched or higher-pitched voices. This can affect individuals across a range of gender identities and vocal characteristics, potentially leading to uneven error rates and impacting overall accessibility.

Following D13, results are computed on the VoxPopuli dev and test sets (jointly) for English (en), French (fr), German (de), Italian (it), and Spanish (es), following "Twists, Humps, and Pebbles: Multilingual Speech Recognition Models Exhibit Gender Performance Gaps", with the addition of two new languages (nl, cs).

The score is the percentage difference between the WER obtained on female and on male speech:

$E(r_{female},\ r_{male})\ =\ 100 \cdot (\phi(r_{female}) - \phi(r_{male})) / \phi(r_{male})$

where $$\phi(\cdot)$$ is the WER metric. This score is named WER_diff(f-m). Ideally it should be 0 — i.e. the model performs equally on the two groups. Positive numbers show that the model performs better on male voices; negative numbers reveal the opposite.

Lang	Qwen2.5-Omni-7B	SpeechLMM-v2.0-L-7B	SpeechLMM-v1.0-L
nl	−7.35	−7.47	—
cs	−6.30	−10.77	—
de	−6.24	−7.13	−0.12
en	−1.54	+0.23	−0.10
es	+5.19	+2.53	+0.01
fr	−5.24	−4.73	−0.10
it	−9.24	−11.03	−0.10
Avg.	−4.39	−5.48	−0.08

SpeechLMM-v2.0-L-7B has a wider average gap than both Qwen2.5-Omni-7B and SpeechLMM-v1.0-L: the average WER_diff(f-m) is −5.48, against −4.39 for the backbone and −0.08 for SpeechLMM-v1.0-L, so the model tends to recognise female voices with a lower error rate than male voices. English is close to parity; Italian and Czech show the widest gaps.

Hallucination and Groundedness

Knowledge-intensive tasks require that the model outputs factually accurate information. This capability is evaluated on two tasks, SQA and SSUM, using the LettuceDetect library to determine whether a given output contains hallucinations. A text is considered a hallucination when its confidence score is ≥ 0.9. The model used for predictions is lettucedect-large-modernbert-en-v1.

Since the hallucination metric is an accuracy, the verbosity of the output strongly impacts the score. In the extreme case where no output is given, the hallucination score is the best (0.0), but the quality of such output is likely very low. Hence, for a fair comparison, the quality score achieved by each model is reported alongside.

Hallucinations in Question Answering: LibriSQA

A subset of LibriSQA – part I, consisting of 250 document/question/answer triplets, is used. The ratio of hallucinated responses over all triplets is reported.

LibriSQA – part I	Average Hallucinations ↓	BertScore ↑
Qwen2.5-Omni-7B	0.392	51.80
SpeechLMM-v1.0-L	n.a.	n.a.
SpeechLMM-v2.0-L-7B	0.444	58.28

Hallucinations in Summarisation: ICSI

For SSUM, the ICSI dataset (6 documents) is used. The average number of hallucinated token spans in the model's output across the whole evaluation portion of ICSI is reported.

Qwen2.5-Omni-7B and SpeechLMM-v2.0-L-7B are evaluated on the same ICSI audio files using identical 240-second audio chunks, prompts, and generation settings. Note that this evaluation does not use the official inference repo; hypotheses are decoded with a custom script that calls model.generate(). Each model summarises every chunk independently; the chronological chunk summaries are then combined into one hypothesis per meeting and scored with the same hallucination-detection evaluator.

ICSI	Average Hallucinations ↓	ROUGE-F1 ↑
Qwen2.5-Omni-7B	14.833	1.84
SpeechLMM-v1.0-L	1.5	n.a.
SpeechLMM-v2.0-L-7B	15.66	6.59

On both probes, fine-tuning improves overlap with the reference and leaves the hallucination rate at or above the level of the backbone.

Language and Data Distribution Bias

Code Switching

Code switching refers to the phenomenon of a speaker switching between two or more languages while speaking. While this is a common phenomenon due to increased globalisation and the prevalence of bilingual communities, today's models still are not capable of reliably handling and transcribing such speech. In many cases such code-switched words happen to contain important socio-linguistic information, which can lead to significant performance drops in downstream tasks based on speech transcriptions.

There are two major categories of code-switching: switching of languages between utterances, and the more challenging intra-sentential code-switching, which describes sparse word insertions of a second language inside otherwise monolingual speech. In some cases there is even the possibility of intra-word code-switching, by morphologically adjusting the foreign-language word to the main language's grammar. In intra-sentential code-switching, the dominant language is called the matrix language, and the inserted language the embedded language.

Evaluation uses standard ASR metrics (WER, CER). Since WER and CER do not capture how well a model transcribes the embedded language, which is often underrepresented in the utterances, PIER [1] is also reported — a dedicated metric focusing on the model's ability to accurately predict embedded-language words.

Some models occasionally generate additional conversational or hallucinated text after the actual transcription. To avoid penalising the ASR score for such model-specific output formatting, simple rule-based post-processing is applied before evaluation. For Qwen, outputs are often followed by conversational phrases such as "if you have any other ..."; all text following this phrase is discarded. SpeechLMM-v1 exhibits similar behaviour on DECM, where hallucinated content frequently appears after the first line break; for this dataset only the text preceding the first line break is retained. For SpeechLMM-v2, hallucinated conversational content is commonly introduced after the token "human"; for all evaluated datasets, text following this token is removed before computing the ASR metrics.

WER / CER / PIER:

Dataset	Langs	Qwen2.5-Omni-7B	Phi-4	SpeechLMM-v1.0-L	SpeechLMM-v2.0-L-7B
DECM	de-en	103.31 / 73.1 / 73.99	45.78 / 30.26 / 58.72	en: 549.5 / 398.2 / 406.88 de: 148.6 / 136.5 / 195.18	95.71 / 65.30 / 68.21
CSFleurs	de-en	84.90 / 62.21 / 48.21	43.77 / 30.28 / 74.78	en: 46 / 27.8 / 60.45 de: 51.3 / 26.1 / 64.4	45.34 / 29.13 / 59.55
Fisher	es-en	91.28 / 76.84 / 114.45	77.57 / 53.86 / 142.09	en: 100.2 / 70.4 / 108.87 es: 76.1 / 54 / 126.16	60.84 / 45.65 / 63.16
Bangor Miami	es-en	112.17 / 96.91 / 116.63	86.81 / 62.66 / 106.97	en: 98.5 / 74 / 75.96 es: 83 / 61.6 / 90.06	77.91 / 58.77 / 91.46
CSFleurs	it-en	85.28 / 62.86 / -	30.41 / 22.66 / -	en: 28.5 / 15.4 / - it: 26.4 / 14.6 / -	28.67 / 16.87 / -

On German–English, Spanish–English, and Italian–English code-switched sets, SpeechLMM-v2.0-L-7B lowers WER relative to Qwen2.5-Omni-7B. Absolute error rates remain high, and PIER shows that words from the embedded language are still poorly transcribed. Phi-4 is stronger on the German–English sets (DECM and CSFleurs); SpeechLMM-v2.0-L-7B is ahead of Phi-4 on the Spanish–English Fisher and Bangor Miami sets.

[1] Ugan, Enes Yavuz, et al. "PIER: A novel metric for evaluating what matters in code-switching." ICASSP 2025 — IEEE International Conference on Acoustics, Speech and Signal Processing. IEEE, 2025.

Accented Speech Recognition

Accented speech refers to speech that exhibits systematic phonetic and phonological characteristics associated with a speaker's regional or linguistic background. In practice, users often speak with a wide range of accents, which can pose challenges for ASR models trained predominantly on standard or mainstream pronunciations.

Evaluation uses WER and CER on the following datasets:

nigerian_accented_english_dataset: Nigerian-accented English speech collected to support low-resource accented ASR research.
Common Voice 14.0 (English): all test-set samples that have an accent tag, where the number of samples for that accent exceeds 10.
Common Voice 14.0 (French): same filtering criteria.

The same rule-based post-processing as above is applied. Qwen frequently appends conversational phrases beginning with "if you have any other ..."; all text following this phrase is discarded. For French audio, Qwen additionally produces recurring Chinese characters with the same fixed meaning across outputs, indicating a systematic generation artefact unrelated to the spoken content; these characters are removed before evaluation. SpeechLMM-v2 tends to append conversational content following the token "human"; all text following this token is discarded.

WER / CER:

Lang	Accent	Qwen2.5-Omni-7B	Phi-4	SpeechLMM-v1.0-L	SpeechLMM-v2.0-L-7B
en	Nigerian	42.06 / 32.98	36.2 / 28.9	71.1 / 48.2	34.28 / 25.28
en	Australian (cv14)	40.35 / 31.32	7.93 / 3.61	9.3 / 4.3	7.24 / 3.55
en	Canadian (cv14)	11.97 / 9.86	6.67 / 3.27	8 / 3.6	6.08 / 3.08
en	England (cv14)	23.51 / 19.25	7.9 / 3.5	8.9 / 3.8	9.88 / 5.03
en	Filipino (cv14)	9.43 / 6.62	5.6 / 1.6	6.6 / 2.2	3.77 / 1.89
en	Hong Kong (cv14)	11.31 / 7.16	8.3 / 3.7	7.1 / 3.9	10.71 / 4.77
en	South Asia (cv14)	28.9 / 24.08	9 / 4.4	9.5 / 4.5	9.82 / 5.17
en	Irish (cv14)	31.55 / 25.43	8 / 4	8.6 / 3.1	16.58 / 8.21
en	Malaysia (cv14)	14.89 / 14.31	6.4 / 2.4	8.5 / 5	4.26 / 1.67
en	New Zealand (cv14)	10.46 / 7.7	3.3 / 1.3	7.8 / 2	1.31 / 0.12
en	Scottish (cv14)	12 / 13.8	2 / 0.9	5.3 / 2.8	4 / 2.13
en	South Africa (cv14)	30.81 / 22.16	9.9 / 6.8	11.4 / 6.1	11.62 / 7.5
en	US (cv14)	29.46 / 24.67	7.7 / 3.8	8.4 / 3.9	7.74 / 4.2
fr	Algerian (cv14)	29.17 / 15.49	7.3 / 3.1	15.6 / 6	17.19 / 7.23
fr	German (cv14)	27.35 / 16.06	14.7 / 6.3	16.3 / 5.8	12.65 / 6.07
fr	Belgium (cv14)	14.53 / 11.29	8.2 / 2.8	10.7 / 3.8	7.38 / 3.15
fr	Ivory Coast (cv14)	15.98 / 5.71	8.8 / 3.1	16.5 / 6.1	11.86 / 6.27
fr	French (cv14)	15.68 / 6.1	9.5 / 3.5	12.2 / 4.6	9.51 / 3.74
fr	US (cv14)	68.37 / 39.18	33.7 / 18	26.5 / 14	30.61 / 16.14
fr	Swiss (cv14)	16.24 / 4.47	9 / 2.6	13.7 / 3.6	8.55 / 2.13
fr	Cameroon (cv14)	56.93 / 32.52	32.1 / 16.8	25.5 / 14.4	36.5 / 21.05
fr	Canada (cv14)	24.40 / 11.9	15.5 / 6.9	17.9 / 6.9	15.39 / 8.17
fr	Morocco (cv14)	12.4 / 3.4	11 / 2.8	10.2 / 2.4	8.76 / 2.46
fr	Senegal (cv14)	13.27 / 6.19	8.9 / 4.4	12.4 / 5.1	17.7 / 9.52

On Nigerian-accented English and on Common Voice 14 English and French accents with more than ten test utterances, SpeechLMM-v2.0-L-7B substantially lowers WER relative to Qwen2.5-Omni-7B and is competitive with Phi-4 and SpeechLMM-v1.0-L on many varieties. Coverage is uneven: Irish English and several French varieties, including Cameroonian, Senegalese, and US-accented French, remain clearly harder than the best-recognised accents in the same collection.

Recommendations

Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. In particular:

Keep a human in the loop for any transcript, summary, or set of minutes that will be acted on; the groundedness probes show ungrounded spans at a rate comparable to the Qwen2.5-Omni-7B backbone.
Do not assume uniform quality across speakers: the measured gender gap is wider than that of the backbone and of SpeechLMM-v1.0-L, and accent coverage is uneven.
Expect degraded transcription on code-switched speech, especially for embedded-language words.
Treat lip-reading output as an auxiliary signal, not as a transcript.

How to Get Started with the Model

See Model Sources for the training, inference and evaluation repositories, and Prompts for the task prompts used at inference.

Training Details

Training Data

TASK	Dataset	Language	License
ACHAP	AMI Meeting Corpus	en → en	CC-BY-4.0
	YTSeg	en → en	CC-BY-NC-SA-4.0
ASR	CoVoST 2	cs → cs, de → de, en → en, es → es, fr → fr, hu → hu, it → it, nl → nl, pt → pt, sv → sv	CC0-1.0
	FLEURS	cs → cs, de → de, en → en, es → es, fr → fr, hu → hu, it → it, nl → nl, pt → pt, sv → sv	CC-BY-4.0
	Multilingual Spoken Topical-Chat	cs → cs, de → de, en → en, es → es, fr → fr, hu → hu, it → it, nl → nl, pt → pt, sv → sv	CDLA-Sharing-1.0
	Spoken DGT-TM	cs → cs, de → de, en → en, es → es, fr → fr, hu → hu, it → it, nl → nl, pt → pt, sv → sv	EUPL
	VoxPopuli	cs → cs, de → de, en → en, es → es, fr → fr, hu → hu, it → it, nl → nl	CC0-1.0
	EuroSpeech	pt → pt, sv → sv	Mixed Non-commercial
	YTSeg	en → en	CC-BY-NC-SA-4.0
AVSPEAKD	AMI Meeting Corpus	en → en	CC-BY-4.0
LIPREAD	LipCrops	en → en	-
MT	Europarl-ST	{de, en, es, fr, it, nl, pt} → {de, en, es, fr, it, nl, pt}	CC-BY-NC-4.0
SLU-I	SLURP – Real & Synth	en → en	CC-BY-NC-4.0
	Speech-MASSIVE	de → de, fr → fr	CC-BY-NC-SA-4.0
SQA-A	LibriSQA – Part I – for IWSLT26	en → en, en → de, en → it	-
SQA-E	Multilingual Spoken SQuAD	cs → cs, de → de, en → en, es → es, fr → fr, hu → hu, it → it, nl → nl, pt → pt, sv → sv	CC-BY-SA-4.0
SSUM	ICSI	en → en	CC-BY-4.0
ST	Europarl-ST	{de, en, es, fr, it, nl, pt} → {de, en, es, fr, it, nl, pt}	CC-BY-NC-4.0
	Multilingual Spoken Topical-Chat	en → {cs, de, es, fr, hu, it, nl, pt, sv}, {cs, de, es, fr, hu, it, nl, pt, sv} → en	CDLA-Sharing-1.0
	Spoken DGT-TM	en → {cs, de, es, fr, hu, it, nl, pt, sv}, {cs, de, es, fr, hu, it, nl, pt, sv} → en	EUPL
TQA-M	Multilingual Text LibriSQA – Part II	cs → cs, de → de, en → en, es → es, fr → fr, hu → hu, it → it, nl → nl, pt → pt, sv → sv	-
TSUM	ICSI	en → en	CC-BY-4.0

There is no dedicated training data for TTS and S2ST. SpeechLMM-v2.0-L-7B reuses the Talker of Qwen2.5-Omni-7B unchanged: it stays frozen in every training stage and is never fine-tuned, so speech output comes entirely from the backbone.

Training Procedure

Training is organised in three stages. Each stage starts from the previous checkpoint. The Talker is frozen throughout and is never fine-tuned.

Stage 0A — Lip-reading adapter. Only the lip-reading adapter is trained, on English lip-crop video, with the lip-reading encoder frozen. This aligns visual speech features with the language model.

Stage 0B — Audio adapter. The audio adapter is fully fine-tuned on all audio tasks and languages. The audio encoder and language model stay frozen. This aligns audio speech features with the language model for the relevant domains and languages.

Stage 1 — Multitask LoRA. The main run continues full fine-tuning of the audio adapter and applies LoRA to the audio encoder and language model. Training covers all text, audio and video tasks, including audio-visual speaker diarisation. Lip-reading data is left out; the video path, lip-reading path, and Talker stay frozen.

Evaluation

Testing Data, Factors & Metrics

For evaluation across all tasks, languages, and datasets, small datasets are used so that the comparison stays manageable. Each small set contains 250 samples per language direction. These subsets match the full sets on some representative tasks (ASR, MT, ST): the mean is identical, and the difference in variance is negligible. The only evaluation set that is not a small subset is ICSI, which contains 6 samples.

Metrics are the standard ones for each benchmark: word and character error rate (WER, CER), BLEU, BLEURT, COMET, intent accuracy, exact match and token F1, and ROUGE. BLEURT and COMET are reported on a 0–100 scale. For S2ST and TTS, metrics prefixed with ASR- are computed on a Whisper large-v3 transcript of the generated speech.

Results below compare SpeechLMM-v2.0-L-7B with its Qwen2.5-Omni-7B backbone.

Results

ACHAP — Audio Chaptering

ACHAP is scored only on output that follows a fixed format — "MM:SS - chapter_title", one line per chapter. Qwen2.5-Omni-7B is not fine-tuned for this task and did not produce that format for most ICSI samples, so no score could be computed.

ICSI	Source Lang	Target Lang	Collar F1 ↑	GC-BS ↑	TM-BS ↑	tm_matched ↑
Qwen2.5-Omni-7B	en	en	-	-	-	-
SpeechLMM-v2.0-L-7B	en	en	2.62	71.73	91.68	27.80

ASR — Automatic Speech Recognition

ACL6060	Source Language	Target Language	WER ↓	CER ↓
Qwen2.5-Omni-7B	en	en	12.38	7.03
SpeechLMM-v2.0-L-7B	en	en	14.19	7.56

CoVoST 2	Source Language	Target Language	WER ↓	CER ↓
Qwen2.5-Omni-7B	de	de	6.06	2.49
SpeechLMM-v2.0-L-7B	de	de	5.55	2.26

Qwen2.5-Omni-7B	en	en	7.35	3.87
SpeechLMM-v2.0-L-7B	en	en	7.75	3.88

Qwen2.5-Omni-7B	es	es	4.69	2.05
SpeechLMM-v2.0-L-7B	es	es	3.92	1.54

Qwen2.5-Omni-7B	fr	fr	7.19	2.42
SpeechLMM-v2.0-L-7B	fr	fr	7.06	2.53

Qwen2.5-Omni-7B	it	it	4.55	1.55
SpeechLMM-v2.0-L-7B	it	it	4.59	1.42

DiPCo	Source Language	Target Language	WER ↓	CER ↓
Qwen2.5-Omni-7B	en	en	25.73	22.51
SpeechLMM-v2.0-L-7B	en	en	21.93	18.27

FLEURS	Source Language	Target Language	WER ↓	CER ↓
Qwen2.5-Omni-7B	cs	cs	93.75	67.52
SpeechLMM-v2.0-L-7B	cs	cs	26.05	11.43

Qwen2.5-Omni-7B	de	de	6.72	3.87
SpeechLMM-v2.0-L-7B	de	de	5.02	2.06

Qwen2.5-Omni-7B	en	en	4.66	2.16
SpeechLMM-v2.0-L-7B	en	en	4.24	1.73

Qwen2.5-Omni-7B	es	es	5.49	2.97
SpeechLMM-v2.0-L-7B	es	es	3.09	1.27

Qwen2.5-Omni-7B	fr	fr	10.47	4.53
SpeechLMM-v2.0-L-7B	fr	fr	7.71	2.43

Qwen2.5-Omni-7B	hu	hu	228.19	202.65
SpeechLMM-v2.0-L-7B	hu	hu	43.68	22.59

Qwen2.5-Omni-7B	it	it	4.69	2.45
SpeechLMM-v2.0-L-7B	it	it	3.38	1.22

Qwen2.5-Omni-7B	nl	nl	11.09	5.10
SpeechLMM-v2.0-L-7B	nl	nl	5.73	2.19

Qwen2.5-Omni-7B	pt	pt	7.65	4.07
SpeechLMM-v2.0-L-7B	pt	pt	3.52	1.40

Qwen2.5-Omni-7B	sv	sv	83.80	54.72
SpeechLMM-v2.0-L-7B	sv	sv	19.82	8.13

Multilingual LibriSpeech	Source Language	Target Language	WER ↓	CER ↓
Qwen2.5-Omni-7B	de	de	7.51	3.04
SpeechLMM-v2.0-L-7B	de	de	10.75	5.84

Qwen2.5-Omni-7B	en	en	5.74	2.85
SpeechLMM-v2.0-L-7B	en	en	6.37	4.91

Qwen2.5-Omni-7B	es	es	5.39	2.38
SpeechLMM-v2.0-L-7B	es	es	8.05	4.39

Qwen2.5-Omni-7B	fr	fr	5.13	2.25
SpeechLMM-v2.0-L-7B	fr	fr	6.95	3.73

Qwen2.5-Omni-7B	it	it	12.55	3.15
SpeechLMM-v2.0-L-7B	it	it	18.83	9.32

Qwen2.5-Omni-7B	nl	nl	12.26	4.14
SpeechLMM-v2.0-L-7B	nl	nl	11.35	4.37

Qwen2.5-Omni-7B	pt	pt	12.39	5.97
SpeechLMM-v2.0-L-7B	pt	pt	19.66	13.89

AVSPEAKD — Audio-Visual Speaker Diarisation

AVSPEAKD is scored only on output that follows a fixed format — "speaker_id: start_time - end_time", one line per speaker. Qwen2.5-Omni-7B is not fine-tuned for this task and did not produce that format for most VoxMM samples, so no score could be computed.

VoxMM	Source Language	Target Language	F1 ↑	mAP ↑
Qwen2.5-Omni-7B	en	en	-	-
SpeechLMM-v2.0-L-7B	en	en	2.81	1.13

LIPREAD — Lip Reading / Visual Speech Recognition

Lip-reading is not a capability of the backbone Qwen2.5-Omni-7B. Visual speech recognition in SpeechLMM 2.0 comes from a dedicated AutoAVSR encoder and adapter added on top of Qwen-Omni; the backbone’s own vision encoder handles general video, not lip crops. Qwen2.5-Omni-7B therefore has no lip-reading pathway and was not evaluated on these sets.

LipCrops	Source Language	Target Language	WER ↓	CER ↓
Qwen2.5-Omni-7B	en	en	-	-
SpeechLMM-v2.0-L-7B	en	en	66.41	47.14

LRS2	Source Language	Target Language	WER ↓	CER ↓
Qwen2.5-Omni-7B	en	en	-	-
SpeechLMM-v2.0-L-7B	en	en	50.23	33.97

VoxMM	Source Language	Target Language	WER ↓	CER ↓
Qwen2.5-Omni-7B	en	en	-	-
SpeechLMM-v2.0-L-7B	en	en	86.05	61.11

MT — Machine Translation

ACL6060	Source Language	Target Language	BLEU ↑	BLEURT ↑	COMET ↑
Qwen2.5-Omni-7B	en	de	34.88	73.10	83.28
SpeechLMM-v2.0-L-7B	en	de	37.14	74.77	84.24

Qwen2.5-Omni-7B	en	fr	53.19	74.19	84.28
SpeechLMM-v2.0-L-7B	en	fr	39.30	70.50	82.58

Qwen2.5-Omni-7B	en	nl	43.22	77.01	86.04
SpeechLMM-v2.0-L-7B	en	nl	33.97	75.07	85.06

Qwen2.5-Omni-7B	en	pt	54.27	77.78	87.54
SpeechLMM-v2.0-L-7B	en	pt	42.72	75.04	86.22

FLORES+	Source Language	Target Language	BLEU ↑	BLEURT ↑	COMET ↑
Qwen2.5-Omni-7B	en	cs	21.75	73.51	86.00
SpeechLMM-v2.0-L-7B	en	cs	20.73	74.13	85.43

Qwen2.5-Omni-7B	en	de	34.33	75.69	86.90
SpeechLMM-v2.0-L-7B	en	de	35.00	77.12	87.34

Qwen2.5-Omni-7B	en	es	25.13	73.88	85.53
SpeechLMM-v2.0-L-7B	en	es	25.23	73.85	85.40

Qwen2.5-Omni-7B	en	fr	44.65	75.24	87.54
SpeechLMM-v2.0-L-7B	en	fr	38.61	73.50	86.44

Qwen2.5-Omni-7B	en	hu	11.93	73.99	76.88
SpeechLMM-v2.0-L-7B	en	hu	11.39	76.37	78.04

Qwen2.5-Omni-7B	en	it	25.62	76.38	86.90
SpeechLMM-v2.0-L-7B	en	it	24.83	76.13	86.89

Qwen2.5-Omni-7B	en	nl	23.23	73.88	85.93
SpeechLMM-v2.0-L-7B	en	nl	23.32	74.49	86.38

Qwen2.5-Omni-7B	en	pt	46.13	77.35	89.15
SpeechLMM-v2.0-L-7B	en	pt	35.66	75.06	87.80

Qwen2.5-Omni-7B	en	sv	30.70	74.48	86.30
SpeechLMM-v2.0-L-7B	en	sv	27.67	76.05	87.25

Qwen2.5-Omni-7B	cs	en	38.43	77.78	87.97
SpeechLMM-v2.0-L-7B	cs	en	34.97	77.16	87.40

Qwen2.5-Omni-7B	de	en	43.94	79.83	89.43
SpeechLMM-v2.0-L-7B	de	en	40.14	78.74	88.80

Qwen2.5-Omni-7B	es	en	30.07	77.02	87.13
SpeechLMM-v2.0-L-7B	es	en	28.31	76.67	86.87

Qwen2.5-Omni-7B	fr	en	41.83	80.05	89.39
SpeechLMM-v2.0-L-7B	fr	en	37.58	79.31	88.64

Qwen2.5-Omni-7B	hu	en	30.17	74.96	86.77
SpeechLMM-v2.0-L-7B	hu	en	27.05	73.83	85.75

Qwen2.5-Omni-7B	it	en	31.89	77.17	87.61
SpeechLMM-v2.0-L-7B	it	en	30.06	77.08	87.42

Qwen2.5-Omni-7B	nl	en	30.58	76.60	87.31
SpeechLMM-v2.0-L-7B	nl	en	28.77	75.97	86.90

Qwen2.5-Omni-7B	pt	en	47.18	80.39	89.54
SpeechLMM-v2.0-L-7B	pt	en	42.28	79.14	88.74

Qwen2.5-Omni-7B	sv	en	45.97	79.23	89.26
SpeechLMM-v2.0-L-7B	sv	en	41.64	78.19	88.54

SLU-I — Spoken Language Understanding (Intent)

SLURP	Source Language	Target Language	Intent accuracy ↑
Qwen2.5-Omni-7B	en	en	49.6
SpeechLMM-v2.0-L-7B	en	en	78.4

Speech-MASSIVE	Source Language	Target Language	Intent accuracy ↑
Qwen2.5-Omni-7B	de	de	50.4
SpeechLMM-v2.0-L-7B	de	de	81.6

Qwen2.5-Omni-7B	es	es	56.4
SpeechLMM-v2.0-L-7B	es	es	85.2

Qwen2.5-Omni-7B	fr	fr	53.6
SpeechLMM-v2.0-L-7B	fr	fr	84.8

Qwen2.5-Omni-7B	hu	hu	5.2
SpeechLMM-v2.0-L-7B	hu	hu	49.6

Qwen2.5-Omni-7B	nl	nl	52.4
SpeechLMM-v2.0-L-7B	nl	nl	84.4

Qwen2.5-Omni-7B	pt	pt	48.0
SpeechLMM-v2.0-L-7B	pt	pt	78.8

SQA-A — Spoken Question Answering (Abstractive)

LibriSQA – part I	Source Language	Target Language	BertScore ↑
Qwen2.5-Omni-7B	en	en	51.72
SpeechLMM-v2.0-L-7B	en	en	58.39

SQA-E — Spoken Question Answering (Extractive)

SpokenSQuAD	Source Language	Target Language	Exact Match ↑	Token F1 ↑
Qwen2.5-Omni-7B	en	en	60.4	73.93
SpeechLMM-v2.0-L-7B	en	en	79.6	87.32

SQA-M — Spoken Question Answering (Multiple Choice)

MMSU	Source Language	Target Language	Accuracy ↑
Qwen2.5-Omni-7B	en	en	62.0
SpeechLMM-v2.0-L-7B	en	en	60.1

SSUM — Speech Summarisation

ICSI	Source Language	Target Language	R-1 F1 ↑	R-2 F1 ↑	R-L F1 ↑
Qwen2.5-Omni-7B	en	en	1.84	0.13	1.76
SpeechLMM-v2.0-L-7B	en	en	6.72	0.82	6.59

ST — Speech Translation

ACL6060	Source Language	Target Language	BLEU ↑	BLEURT ↑	COMET ↑
Qwen2.5-Omni-7B	en	de	32.27	66.84	78.77
SpeechLMM-v2.0-L-7B	en	de	29.62	69.68	80.16

Qwen2.5-Omni-7B	en	fr	41.24	63.14	78.74
SpeechLMM-v2.0-L-7B	en	fr	36.02	64.06	79.11

Qwen2.5-Omni-7B	en	nl	28.52	66.60	79.56
SpeechLMM-v2.0-L-7B	en	nl	27.67	69.10	80.69

Qwen2.5-Omni-7B	en	pt	35.83	65.87	81.42
SpeechLMM-v2.0-L-7B	en	pt	36.79	67.49	82.03

CoVoST 2	Source Language	Target Language	BLEU ↑	BLEURT ↑	COMET ↑
Qwen2.5-Omni-7B	en	de	31.88	70.80	82.97
SpeechLMM-v2.0-L-7B	en	de	32.50	71.70	82.95

Qwen2.5-Omni-7B	de	en	39.73	75.39	86.12
SpeechLMM-v2.0-L-7B	de	en	38.59	73.62	84.96

Qwen2.5-Omni-7B	es	en	43.66	75.76	86.63
SpeechLMM-v2.0-L-7B	es	en	41.71	75.15	86.59

Qwen2.5-Omni-7B	fr	en	40.12	74.19	84.96
SpeechLMM-v2.0-L-7B	fr	en	40.50	74.17	84.99

Qwen2.5-Omni-7B	it	en	37.88	73.76	84.87
SpeechLMM-v2.0-L-7B	it	en	36.04	73.23	84.90

CVSS	Source Language	Target Language	BLEU ↑	BLEURT ↑	COMET ↑
Qwen2.5-Omni-7B	de	en	21.82	69.67	81.71
SpeechLMM-v2.0-L-7B	de	en	21.32	68.81	80.85

Qwen2.5-Omni-7B	es	en	25.93	74.03	84.61
SpeechLMM-v2.0-L-7B	es	en	25.60	73.80	85.04

Qwen2.5-Omni-7B	fr	en	23.73	70.88	81.60
SpeechLMM-v2.0-L-7B	fr	en	23.08	71.34	81.42

Qwen2.5-Omni-7B	it	en	25.30	71.84	83.79
SpeechLMM-v2.0-L-7B	it	en	25.22	71.85	83.43

Qwen2.5-Omni-7B	nl	en	27.93	70.24	82.98
SpeechLMM-v2.0-L-7B	nl	en	26.73	71.69	83.53

Qwen2.5-Omni-7B	pt	en	37.98	76.14	86.64
SpeechLMM-v2.0-L-7B	pt	en	35.75	75.80	86.41

Qwen2.5-Omni-7B	sv	en	1.27	27.68	48.71
SpeechLMM-v2.0-L-7B	sv	en	12.22	53.40	68.81

Europarl-ST	Source Language	Target Language	BLEU ↑	BLEURT ↑	COMET ↑
Qwen2.5-Omni-7B	en	de	25.05	65.91	84.30
SpeechLMM-v2.0-L-7B	en	de	27.85	67.39	86.55

Qwen2.5-Omni-7B	en	es	32.40	44.68	83.07
SpeechLMM-v2.0-L-7B	en	es	37.28	47.24	85.33

Qwen2.5-Omni-7B	en	fr	30.15	46.78	83.93
SpeechLMM-v2.0-L-7B	en	fr	36.36	49.30	87.75

Qwen2.5-Omni-7B	en	it	22.05	46.84	82.84
SpeechLMM-v2.0-L-7B	en	it	23.93	48.62	86.38

Qwen2.5-Omni-7B	en	nl	22.04	64.76	84.62
SpeechLMM-v2.0-L-7B	en	nl	25.25	67.09	86.08

Qwen2.5-Omni-7B	en	pt	28.20	31.76	84.51
SpeechLMM-v2.0-L-7B	en	pt	32.13	34.93	86.95

Qwen2.5-Omni-7B	de	en	14.88	54.06	74.90
SpeechLMM-v2.0-L-7B	de	en	34.34	61.40	84.81

Qwen2.5-Omni-7B	es	en	9.03	52.26	68.60
SpeechLMM-v2.0-L-7B	es	en	39.10	63.67	82.65

Qwen2.5-Omni-7B	fr	en	18.89	56.07	74.98
SpeechLMM-v2.0-L-7B	fr	en	40.08	63.71	84.62

Qwen2.5-Omni-7B	it	en	20.23	57.50	74.85
SpeechLMM-v2.0-L-7B	it	en	37.61	65.79	83.40

Qwen2.5-Omni-7B	nl	en	19.20	54.28	74.92
SpeechLMM-v2.0-L-7B	nl	en	35.25	61.90	84.41

Qwen2.5-Omni-7B	pt	en	24.86	57.91	75.82
SpeechLMM-v2.0-L-7B	pt	en	40.05	64.97	84.75

FLEURS+	Source Language	Target Language	BLEU ↑	BLEURT ↑	COMET ↑
Qwen2.5-Omni-7B	en	cs	15.67	68.09	81.09
SpeechLMM-v2.0-L-7B	en	cs	18.50	70.95	83.66

Qwen2.5-Omni-7B	en	de	29.92	74.82	86.27
SpeechLMM-v2.0-L-7B	en	de	30.49	75.26	86.26

Qwen2.5-Omni-7B	en	es	23.79	72.02	84.14
SpeechLMM-v2.0-L-7B	en	es	24.58	73.63	85.32

Qwen2.5-Omni-7B	en	fr	36.28	70.36	84.19
SpeechLMM-v2.0-L-7B	en	fr	37.71	72.84	86.14

Qwen2.5-Omni-7B	en	hu	7.72	73.62	75.80
SpeechLMM-v2.0-L-7B	en	hu	9.38	76.23	77.88

Qwen2.5-Omni-7B	en	it	22.83	74.03	85.01
SpeechLMM-v2.0-L-7B	en	it	24.36	74.92	85.71

Qwen2.5-Omni-7B	en	nl	21.21	71.03	83.95
SpeechLMM-v2.0-L-7B	en	nl	22.17	72.95	85.66

Qwen2.5-Omni-7B	en	pt	38.35	74.38	87.10
SpeechLMM-v2.0-L-7B	en	pt	34.57	74.31	87.57

Qwen2.5-Omni-7B	en	sv	27.63	69.40	82.33
SpeechLMM-v2.0-L-7B	en	sv	29.25	73.43	85.11

Qwen2.5-Omni-7B	cs	en	3.99	40.98	57.55
SpeechLMM-v2.0-L-7B	cs	en	21.63	65.45	78.90

Qwen2.5-Omni-7B	de	en	37.51	77.73	87.74
SpeechLMM-v2.0-L-7B	de	en	38.01	77.71	87.77

Qwen2.5-Omni-7B	es	en	25.93	75.22	85.76
SpeechLMM-v2.0-L-7B	es	en	28.42	75.98	86.54

Qwen2.5-Omni-7B	fr	en	35.76	77.16	87.39
SpeechLMM-v2.0-L-7B	fr	en	35.32	77.32	87.55

Qwen2.5-Omni-7B	hu	en	0.34	27.95	44.75
SpeechLMM-v2.0-L-7B	hu	en	12.76	51.73	68.82

Qwen2.5-Omni-7B	it	en	27.97	75.53	86.21
SpeechLMM-v2.0-L-7B	it	en	29.46	75.76	86.46

Qwen2.5-Omni-7B	nl	en	27.13	73.81	85.00
SpeechLMM-v2.0-L-7B	nl	en	28.46	74.49	85.91

Qwen2.5-Omni-7B	pt	en	42.56	77.94	87.81
SpeechLMM-v2.0-L-7B	pt	en	42.11	78.33	88.21

Qwen2.5-Omni-7B	sv	en	6.35	38.49	54.16
SpeechLMM-v2.0-L-7B	sv	en	29.43	68.23	81.22

S2ST — Speech-to-Speech Translation

ASR-BLEU, ASR-BLEURT and ASR-COMET are the ST metrics computed on a Whisper large-v3 transcript of the generated speech, scored against the text reference.

Note on speech output. SpeechLMM-v2.0-L-7B reuses the Talker of Qwen2.5-Omni-7B unchanged: it is frozen in every training stage and is never fine-tuned, so the speech scored below is produced by the backbone Talker. That Talker produces reliable speech in English and Chinese only. Hence, only directions into English are reported below.

CVSS	Source Language	Target Language	ASR-BLEU ↑	ASR-BLEURT ↑	ASR-COMET ↑
Qwen2.5-Omni-7B	de	en	22.36	69.27	81.22
SpeechLMM-v2.0-L-7B	de	en	22.39	65.92	78.98

Qwen2.5-Omni-7B	es	en	26.89	73.62	83.93
SpeechLMM-v2.0-L-7B	es	en	24.46	70.38	81.56

Qwen2.5-Omni-7B	fr	en	22.16	68.30	78.17
SpeechLMM-v2.0-L-7B	fr	en	21.09	63.73	74.90

Qwen2.5-Omni-7B	it	en	24.24	71.49	82.78
SpeechLMM-v2.0-L-7B	it	en	21.79	68.67	80.22

Qwen2.5-Omni-7B	nl	en	24.14	71.32	83.39
SpeechLMM-v2.0-L-7B	nl	en	22.73	71.61	83.61

Qwen2.5-Omni-7B	pt	en	35.76	76.92	86.27
SpeechLMM-v2.0-L-7B	pt	en	34.75	73.88	84.62

Qwen2.5-Omni-7B	sv	en	0.66	27.98	46.63
SpeechLMM-v2.0-L-7B	sv	en	9.32	49.19	65.60

FLEURS+	Source Language	Target Language	ASR-BLEU ↑	ASR-BLEURT ↑	ASR-COMET ↑
Qwen2.5-Omni-7B	cs	en	4.23	38.60	54.42
SpeechLMM-v2.0-L-7B	cs	en	19.77	59.01	72.98

Qwen2.5-Omni-7B	de	en	36.87	76.48	86.98
SpeechLMM-v2.0-L-7B	de	en	32.48	69.64	81.60

Qwen2.5-Omni-7B	es	en	24.91	74.25	85.18
SpeechLMM-v2.0-L-7B	es	en	24.14	64.95	77.57

Qwen2.5-Omni-7B	fr	en	33.06	75.29	85.94
SpeechLMM-v2.0-L-7B	fr	en	29.49	68.13	79.75

Qwen2.5-Omni-7B	hu	en	0.25	26.11	43.30
SpeechLMM-v2.0-L-7B	hu	en	9.24	45.60	62.48

Qwen2.5-Omni-7B	it	en	26.69	74.84	85.66
SpeechLMM-v2.0-L-7B	it	en	26.03	69.61	81.35

Qwen2.5-Omni-7B	nl	en	25.48	71.69	83.88
SpeechLMM-v2.0-L-7B	nl	en	23.61	64.18	77.62

Qwen2.5-Omni-7B	pt	en	40.12	75.72	86.61
SpeechLMM-v2.0-L-7B	pt	en	32.06	65.29	77.41

Qwen2.5-Omni-7B	sv	en	4.51	37.86	53.48
SpeechLMM-v2.0-L-7B	sv	en	21.38	58.65	73.25

TSUM — Text Summarisation

ICSI	Source Language	Target Language	R-1 F1 ↑	R-2 F1 ↑	R-L F1 ↑
Qwen2.5-Omni-7B	en	en	24.72	4.05	22.25
SpeechLMM-v2.0-L-7B	en	en	19.80	3.32	17.18

TTS — Text-to-Speech

ASR-WER and ASR-CER are computed using a Whisper large-v3 transcript of the generated speech against the input text.

Note on speech output. SpeechLMM-v2.0-L-7B reuses the Talker of Qwen2.5-Omni-7B unchanged: it is frozen in every training stage and is never fine-tuned, so the speech scored below is produced by the backbone Talker. That Talker produces reliable speech in English and Chinese only. Hence, only directions into English are reported below.

LJSpeech	Source Language	Target Language	ASR-WER ↓	ASR-CER ↓	UTMOS ↑
Qwen2.5-Omni-7B	en	en	4.93	4.00	4.29
SpeechLMM-v2.0-L-7B	en	en	17.40	15.63	4.30

Summary

Against Qwen2.5-Omni-7B, fine-tuning brings the largest speech-recognition gains on FLEURS, including Czech, Hungarian, and Swedish, where the backbone word error rate is very high, and raises spoken-intent accuracy on SLURP and Speech-MASSIVE on all evaluated languages. Extractive spoken question answering on SpokenSQuAD, and speech translation on Europarl-ST and on several lower-resourced FLEURS+ directions, also improve. Audio chaptering and meeting summarisation improve on ICSI, while the absolute scores stay low. Machine translation, multiple-choice spoken question answering (MMSU), and several high-resource recognition sets (Multilingual LibriSpeech, ACL6060) are mixed, and the backbone remains ahead on a number of directions. Text-to-speech on LJSpeech degrades relative to the backbone in ASR-WER/ASR-CER while UTMOS is comparable. Lip-reading word error rate remains far above a usable range on all three visual-speech sets (LipCrops 66.41, LRS2 50.23, VoxMM 86.05).

Prompts

The prompts used for inference and training, per task.

Task	Inference prompt example	Training prompt template
ACHAP	Given the English speech input, identify the chapters. Return one chapter per line in the format: MM:SS - chapter_title. Use HH:MM:SS for timestamps of one hour or longer. Each timestamp must indicate the chapter's start within the audio. Write chapter titles in English and return them in chronological order. Return only the timestamp and title lines, without a transcript, Markdown headings, or additional commentary.	Given the {lang_in} {format_in} input, identify the chapters. For each chapter, return its start time and title in this format: 'start_time - chapter_title'. Chapter titles must be written in {lang_out}. Return the chapters in chronological order. Input: {input}
ASR	Transcribe the German speech input. The transcription must be written in German. Return only the transcription.	Transcribe the {lang_in} {format_in} input. The transcription must be written in {lang_in} Input: {input}
LIPREAD	Transcribe the speech visible in the English video input. The transcription must be written in English.	Transcribe the speech visible in the {lang_in} {format_in} input. The transcription must be written in {lang_in}. Input: {input}
MT	Translate the Czech text input into English text. Preserve the meaning of the original input. Return only the translation.	Translate the {lang_in} {format_in} input into {lang_out} {format_out}. Preserve the meaning of the original input. Input: {input}
SLU-I	Given the German speech, identify the speaker's intent. Choose exactly one intent from this list: alarm_query, alarm_remove, alarm_set, audio_volume_down, … weather_query, wemo_off. If the intent is not present, reply with 'unanswerable'. Return only the exact intent label. (the intent list is defined by the dataset)	Given the {lang_in} {format_in}, identify the speaker's intent. Choose exactly one intent from the following intent list. If the intent is not present in the list, reply with 'unanswerable'. Input: {input} Intent list: {intents}
SQA-A	Given the English speech input, answer the accompanying question. Base the answer only on information contained in the input. Answer in English, concisely but with all requested details. Return only the answer.	Given the {lang_in} {format_in} input, answer the {lang_qst} {format_qst} question. Base the answer only on the information contained in the input. The answer must be written in {lang_out}. Input: {input} Question: {question}
SQA-E	Given the English speech input, answer the accompanying question. Extract the exact words from the input that answer the question. Do not paraphrase or add words. The answer must be written in English. If absent, reply with 'unanswerable'.	Given the {lang_in} {format_in} input, answer the {lang_qst} {format_qst} question. Extract the exact words from the input that answer the question. Do not paraphrase, modify, or add any words. The answer must be written in {lang_out}. If the answer is not present in the input, reply with 'unanswerable'. Input: {input} Question: {question}
SQA-M	Given the input, answer the text question. Choose exactly one answer from the following options. Return only the answer label: A, B, C, or D. (the actual answer choices are given within the input)	Given the {lang_in} {format_in} input, answer the {lang_qst} {format_qst} question. Choose exactly one answer from the following options. Input: {input} Question: {question} Options: {options}
SSUM	Summarize the following English speech input. Write the summary in English. Return only the summary.	Summarize the {lang_in} {format_in} input. The summary must be written in {lang_out}. Input: {input}
ST	Translate the German speech input into English text. Preserve the meaning of the original input. Return only the translation.	Translate the {lang_in} {format_in} input into {lang_out} {format_out}. Preserve the meaning of the original input. Input: {input}
S2ST	Translate the Czech speech input into English speech. Preserve the meaning of the original input. Return only the translation, and then stop.	—
TSUM	Summarize the following English text input. Write the summary in English. Return only the summary.	Summarize the {lang_in} {format_in} input. The summary must be written in {lang_out}. Input: {input}
TTS	Generate English speech from the English text input.\nSpeak only the text input, and then stop.	—
ASR (Gender Bias)	Transcribe the German speech input. The transcription must be written in German. Return only the transcription.	Transcribe the {lang_in} {format_in} input. The transcription must be written in {lang_in} Input: {input}
Hallucinations in QA	Given the English speech input, answer the accompanying question. Extract the exact words from the input that answer the question. Do not paraphrase or add words. The answer must be written in English. If absent, reply with 'unanswerable'.	Given the {lang_in} {format_in} input, answer the {lang_qst} {format_qst} question. Extract the exact words from the input that answer the question. Do not paraphrase, modify, or add any words. The answer must be written in {lang_out}. If the answer is not present in the input, reply with 'unanswerable'. Input: {input} Question: {question}
Hallucinations in Summarisation	Summarize the English speech input. The summary must be written in English.	Summarize the {lang_in} speech input. The summary must be written in {lang_in}. {input}:
Code Switching	Transcribe the German, English speech input. The transcription must be written in German, English. Input:	Transcribe the {lang_in} speech input. The transcription must be written in {lang_in}. Input: — {lang_in} ∈ {German, English; Spanish, English; Italian, English}
Accented Speech Recognition	Transcribe the English speech input. The transcription must be written in English. Input:	Transcribe the {lang_in} speech input. The transcription must be written in {lang_in}. Input: — {lang_in} ∈ {English, French}

Environmental Impact

Carbon emissions can be estimated using the Machine Learning Impact calculator presented in Lacoste et al. (2019).

Technical Specifications

Model Architecture and Objective

SpeechLMM-v2.0-L-7B uses Qwen2.5-Omni-7B as the multimodal backbone. Qwen2.5-Omni is a Thinker–Talker model that unifies perception, understanding and generation across text, image, audio, and video:

The Thinker handles multimodal reasoning and text generation (~7B parameters).
The Talker produces streaming speech tokens conditioned on hidden states from the Thinker (used for TTS and speech-to-speech translation).
The audio encoder is initialised from Whisper-large-v3 and processes audio in blocks, so long inputs can be encoded without rebuilding alignment to the language model.
The vision encoder is the ViT from Qwen2.5-VL. In SpeechLMM it is used for general video (for example audio-visual speaker diarisation), not for lip-reading.
The lip-reading encoder is AutoAVSR, used for visual speech / lip crops.

The backbone already covers a wide set of written and spoken languages. SpeechLMM still trains for ten EU languages, including Czech, Hungarian, and Swedish. Updating the audio encoder and adapter (Stages 0B and 1) is how those languages are brought into the speech-understanding path.

On top of Qwen2.5-Omni, SpeechLMM v2 adds the pieces needed for meeting scenarios:

Component	Role
Backbone	Qwen2.5-Omni-7B (Thinker + Talker + Whisper-based audio encoder + Qwen2.5-VL vision encoder)
Lip-reading encoder	AutoAVSR for visual speech / lip crops
Adapters	Trainable projections from audio, video, and lip-read encoder outputs into Thinker space

The dedicated AutoAVSR pathway is the main architectural addition relative to Qwen2.5-Omni: it supplies visual speech features when audio is degraded or absent, separate from the Qwen2.5-VL vision tower used for general video.

Citation

Referenced works:

Savoldi, B. et al. "Twists, Humps, and Pebbles: Multilingual Speech Recognition Models Exhibit Gender Performance Gaps", EMNLP 2024.
Ugan, E. Y. et al. "PIER: A novel metric for evaluating what matters in code-switching", ICASSP 2025.
Mitchell, M. et al. "Model Cards for Model Reporting", 2018.

Glossary

Abbreviation	Meaning
ACHAP	Audio Chaptering
ASR	Automatic Speech Recognition
AutoAVSR	Automatic Audio-Visual Speech Recognition
AVSPEAKD	Audiovisual Speaker Diarisation
BLEU	Bilingual Evaluation Understudy
CER	Character Error Rate
DST	Downstream Tasks
LLM	Large Language Model
LIPREAD	Lip Reading
LoRA	Low-Rank Adaptation
MT	Machine Translation
PIER	Embedded-language Error Rate (code-switching)
R-1 / R-2 / R-L	ROUGE-1 / ROUGE-2 / ROUGE-L
S2ST	Speech-to-Speech Translation
SLU-I	Spoken Language Understanding – Intent
SQA	Spoken Question Answering
SQA-A	Spoken Question Answering – Abstractive
SQA-E	Spoken Question Answering – Extractive
SQA-M	Spoken Question Answering – Multiple Choice
SSUM	Speech Summarization
ST	Speech Translation
TQA	Textual Question Answering
TQA-A	Textual Question Answering – Abstractive
TQA-E	Textual Question Answering – Extractive
TQA-M	Textual Question Answering – Multiple Choice
TSUM	Text Summarization
TTS	Text-to-Speech
VSR	Visual Speech Recognition
WER	Word Error Rate