Model Card for WavLM-SER (8-class Speech Emotion Recognition)

This model is a fine-tuned 24-layer Transformer (WavLM) for 8-class speech emotion recognition (SER). It was trained on a combination of RAVDESS, IEMOCAP and CREMA-D and reaches 77.3% accuracy in-distribution. Cross-corpus testing showed a 16–23 percentage point drop in accuracy. A failure analysis along the arousal and valence dimensions attributes this to a known limitation of self-supervised speech embeddings, rather than to a bug in the pipeline. See Bias, Risks, and Limitations.

Model Details

Model Description

WavLM is a self-supervised speech representation model. Here, a 24-layer Transformer encoder was fine-tuned with a classification head to predict one of 8 emotion classes from raw speech. The project's main aim was to measure how well SER generalises beyond the corpora it was trained on, and to diagnose why it does not. Planned mitigations (adversarial domain adaptation, speaker disentanglement and a Kenya-specific domain-adaptive pretraining corpus) are scoped but not yet implemented in this release.

  • Developed by: [More Information Needed]
  • Funded by [optional]: [More Information Needed]
  • Shared by [optional]: [More Information Needed]
  • Model type: 24-layer Transformer encoder (WavLM) with a classification head, audio classification
  • Language(s) (NLP): English (acted and elicited emotional speech)
  • License: [More Information Needed]
  • Finetuned from model [optional]: microsoft/wavlm-large [Confirm]

Model Sources [optional]

  • Repository: [More Information Needed]
  • Paper [optional]: [More Information Needed]
  • Demo [optional]: [More Information Needed]

Uses

Direct Use

Research and benchmarking of speech emotion recognition on clean, acted or elicited English speech similar to RAVDESS, IEMOCAP and CREMA-D. It is also suitable as a baseline for studying cross-corpus generalisation and domain shift in SER.

Downstream Use [optional]

Starting point for further work on domain adaptation, speaker-invariant emotion representations, or adaptation to under-represented accents and recording conditions (for example, East African English). It is not intended to be dropped into a production system without re-validation on target-domain data.

Out-of-Scope Use

  • Clinical, diagnostic or mental-health assessment.
  • Hiring, education, insurance, credit, law-enforcement or other decisions that materially affect people.
  • Surveillance or emotion monitoring of people without informed consent.
  • Deployment on speech that differs substantially from the training corpora (different languages, accents, microphones, noise conditions or spontaneous conversation) without target-domain evaluation. Results in this card show that accuracy degrades sharply under such shift.

Bias, Risks, and Limitations

  • Poor cross-corpus generalisation. Accuracy falls by 16–23 pp when evaluated on a corpus not seen in training, compared with in-distribution accuracy of 77.3%.
  • Arousal/valence failure pattern. Failure analysis indicates that errors concentrate where emotions are separable mainly by valence (positive vs. negative with similar arousal), which self-supervised speech embeddings are known to encode less reliably than arousal. Confusions between high-arousal or low-arousal classes are comparatively less severe. [Add your per-class confusion figures here.]
  • Corpus and speaker bias. The training corpora are largely acted or scripted, recorded in studio or lab conditions, and feature mostly North American English speakers. The model may be partly learning speaker, corpus or recording-channel cues in addition to emotion.
  • Under-representation of African English. No Kenyan or wider East African speech is included in training or evaluation. Performance on such speakers is unmeasured and likely lower.
  • Acted vs. felt emotion. Labels reflect perceived or intended emotion in performed speech, not verified internal states.
  • Label-set mismatch. [More Information Needed: describe how the 8 classes were harmonised across the three corpora, since the original label sets differ.]

Recommendations

Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. Evaluate on in-domain data before any use, report cross-corpus or held-out-speaker results rather than only in-distribution accuracy, and do not use outputs to make consequential decisions about individuals.

How to Get Started with the Model

Use the code below to get started with the model.

import torch
import librosa
from transformers import AutoFeatureExtractor, AutoModelForAudioClassification

model_id = "YOUR_USERNAME/YOUR_MODEL_NAME"  # [More Information Needed]

extractor = AutoFeatureExtractor.from_pretrained(model_id)
model = AutoModelForAudioClassification.from_pretrained(model_id).eval()

# WavLM expects 16 kHz mono audio
speech, sr = librosa.load("example.wav", sr=16000, mono=True)
inputs = extractor(speech, sampling_rate=16000, return_tensors="pt", padding=True)

with torch.no_grad():
    logits = model(**inputs).logits

pred_id = logits.argmax(dim=-1).item()
print(model.config.id2label[pred_id])

Training Details

Training Data

Combined training set drawn from three public emotional speech corpora:

  • RAVDESS: acted speech and song by professional actors.
  • IEMOCAP: dyadic, scripted and improvised sessions.
  • CREMA-D: acted sentences from a diverse set of actors.

[More Information Needed: train/validation/test splits, whether splits are speaker-independent, the 8-class label mapping, and the number of utterances per corpus and class.]

Training Procedure

Preprocessing [optional]

[More Information Needed: resampling to 16 kHz, loudness normalisation, max/min clip length, truncation or padding strategy, augmentation.]

Training Hyperparameters

  • Training regime: [More Information Needed]
  • Learning rate / schedule: [More Information Needed]
  • Batch size: [More Information Needed]
  • Epochs: [More Information Needed]
  • Layer freezing / pooling strategy: [More Information Needed]

Speeds, Sizes, Times [optional]

[More Information Needed]

Evaluation

Testing Data, Factors & Metrics

Testing Data

  • In-distribution: held-out test split from the combined RAVDESS + IEMOCAP + CREMA-D data. [More Information Needed: speaker-independent?]
  • Cross-corpus: evaluation on corpora not seen during training. [More Information Needed: specify which corpora and which were held out.]

Factors

Evaluation is disaggregated by corpus (in-distribution vs. cross-corpus) and by affective dimension (arousal and valence), the latter used for failure analysis.

Metrics

Accuracy as the headline metric. [More Information Needed: add unweighted average recall (UAR) and macro-F1 if available, since they are more informative under class imbalance.]

Results

Setting Accuracy
In-distribution (combined test split) 77.3%
Cross-corpus [More Information Needed]: 16–23 pp lower than in-distribution

Summary

The model performs well on data matching its training distribution but does not transfer reliably to unseen corpora. The arousal/valence analysis points to weak valence discrimination as the main source of error, consistent with documented limits of self-supervised speech embeddings for emotion.

Model Examination [optional]

Arousal/valence failure analysis was used to diagnose cross-corpus errors. [More Information Needed: link to confusion matrices, layer-wise probing or other interpretability work.]

Environmental Impact

Carbon emissions can be estimated using the Machine Learning Impact calculator presented in Lacoste et al. (2019).

  • Hardware Type: [More Information Needed]
  • Hours used: [More Information Needed]
  • Cloud Provider: [More Information Needed]
  • Compute Region: [More Information Needed]
  • Carbon Emitted: [More Information Needed]

Technical Specifications [optional]

Model Architecture and Objective

24-layer Transformer encoder (WavLM) with a classification head, trained with cross-entropy loss for 8-way emotion classification.

Compute Infrastructure

[More Information Needed]

Hardware

[More Information Needed]

Software

[More Information Needed: PyTorch, Transformers and other library versions.]

Citation [optional]

If you use this model, please also cite WavLM:

BibTeX:

@article{chen2022wavlm,
  title   = {WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing},
  author  = {Chen, Sanyuan and Wang, Chengyi and Chen, Zhengyang and Wu, Yu and Liu, Shujie and Chen, Zhuo and Li, Jinyu and Kanda, Naoyuki and Yoshioka, Takuya and Xiao, Xiong and others},
  journal = {IEEE Journal of Selected Topics in Signal Processing},
  year    = {2022}
}

[More Information Needed: add a citation for this model or project.]

APA:

[More Information Needed]

Glossary [optional]

  • SER: Speech emotion recognition.
  • Arousal: how activated or energetic an emotion is (calm to excited).
  • Valence: how positive or negative an emotion is.
  • In-distribution: test data from the same corpora as the training data.
  • Cross-corpus: test data from a corpus unseen in training.
  • DAPT: Domain-adaptive pretraining, continued self-supervised pretraining on target-domain data.

More Information [optional]

Planned work (scoped, not implemented):

  1. Adversarial domain adaptation to reduce corpus-specific cues.
  2. Speaker disentanglement to reduce speaker identity leakage into emotion predictions.
  3. A Kenya-specific DAPT corpus to improve robustness to East African English.

Model Card Authors [optional]

[More Information Needed]

Model Card Contact

[More Information Needed]

Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Model tree for space-cadet/wavlm-ser

Finetuned
(40)
this model

Space using space-cadet/wavlm-ser 1

Paper for space-cadet/wavlm-ser