--- library_name: transformers license: cc-by-4.0 base_model: roberta-base metrics: - accuracy - f1 tags: - generated_from_trainer - text-classification - classification - nlp - vulnerability datasets: - CIRCL/vulnerability-scores model-index: - name: vulnerability-severity-classification-roberta-base results: - task: type: text-classification name: Vulnerability severity classification dataset: type: CIRCL/vulnerability-scores name: CIRCL/vulnerability-scores split: test revision: 55a886a483bb27aabb6461edc64e8058cc136e24 metrics: - type: accuracy value: 0.7415 name: Accuracy - type: f1 value: 0.6460 name: F1 Macro args: average: macro --- # VLAI: A RoBERTa-Based Model for Automated Vulnerability Severity Classification ## Severity classification This model is a fine-tuned version of [roberta-base](https://huggingface.co/roberta-base) on the dataset [CIRCL/vulnerability-scores](https://huggingface.co/datasets/CIRCL/vulnerability-scores). The model was presented in the paper [VLAI: A RoBERTa-Based Model for Automated Vulnerability Severity Classification](https://huggingface.co/papers/2507.03607) [[arXiv](https://arxiv.org/abs/2507.03607)]. **Abstract:** VLAI is a transformer-based model that predicts software vulnerability severity levels directly from text descriptions. Built on RoBERTa, VLAI is fine-tuned on over 600,000 real-world vulnerabilities and achieves over 82% accuracy in predicting severity categories, enabling faster and more consistent triage ahead of manual CVSS scoring. The model and dataset are open-source and integrated into the Vulnerability-Lookup service. > **Note (October 2026):** since [VulnTrain 3.4.0](https://github.com/vulnerability-lookup/VulnTrain/releases/tag/v3.4.0), > the dataset is split so that records of the same vulnerability (for example a CVE and its > GitHub advisory) never appear in both train and test. Accuracy measured this way (**74%**) > reflects performance on vulnerabilities the model has never seen. Earlier figures, including > the one in the abstract above, were measured on a split where many test descriptions also > appeared in training. Details in the technical report > [Improving vulnerability-scores: Leakage-Free Splits and Consistent CVSS 4.0 Scoring](https://doi.org/10.5281/zenodo.23760802). You can read [this page](https://www.vulnerability-lookup.org/user-manual/ai/) for more information. ## Model description It is a classification model and is aimed to assist in classifying vulnerabilities by severity based on their descriptions. Labels: | Label | CVSS score | |-------|------------| | Low | < 4.0 | | Medium | 4.0 – 6.9 | | High | 7.0 – 8.9 | | Critical | ≥ 9.0 | ## How to get started with the model ```python from transformers import AutoModelForSequenceClassification, AutoTokenizer import torch model_name = "CIRCL/vulnerability-severity-classification-roberta-base" tokenizer = AutoTokenizer.from_pretrained(model_name) model = AutoModelForSequenceClassification.from_pretrained(model_name) model.eval() print("Model revision:", model.config._commit_hash) test_description = "SAP NetWeaver Visual Composer Metadata Uploader is not protected with a proper authorization, allowing unauthenticated agent to upload potentially malicious executable binaries \ that could severely harm the host system. This could significantly affect the confidentiality, integrity, and availability of the targeted system." inputs = tokenizer(test_description, return_tensors="pt", truncation=True, padding=True) # Run inference with torch.no_grad(): outputs = model(**inputs) predictions = torch.nn.functional.softmax(outputs.logits, dim=-1) # Print results print("Predictions:", predictions) predicted_class = torch.argmax(predictions, dim=-1).item() print("Predicted severity:", model.config.id2label[predicted_class]) ``` ## Training data Revision [`55a886a`](https://huggingface.co/datasets/CIRCL/vulnerability-scores/tree/55a886a483bb27aabb6461edc64e8058cc136e24) of [CIRCL/vulnerability-scores](https://huggingface.co/datasets/CIRCL/vulnerability-scores), built with VulnTrain 3.4.0 from CVE records, GitHub, PySec and CSAF (Red Hat, Cisco, CISA) advisories: | Split | Rows | Labelled rows | |-------|-----:|--------------:| | train | 670,283 | 567,964 | | validation | 83,550 | 71,470 | | test | 83,376 | 70,472 | - **Group-aware split (80/10/10).** Rows that share a description, an identifier or an alias are kept in the same split, so no vulnerability appears in both train and test. - **Labels** come from the first available CVSS score, in the order v4.0, v3.1, v3.0, v2.0. CVSS 4.0 scores are read with base metrics only (CVSS-B), whatever the source. ## Training procedure ### Training hyperparameters The following hyperparameters were used during training: - learning_rate: 3e-05 - train_batch_size: 32 - eval_batch_size: 32 - seed: 42 - optimizer: Use OptimizerNames.ADAMW_TORCH_FUSED with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments - lr_scheduler_type: linear - num_epochs: 5 The checkpoint with the best validation accuracy (epoch 4) is the one published. Training took 4 h 25 min on 4 × NVIDIA L40S and used 4.61 kWh, for 0.48 kg CO₂eq (measured with CodeCarbon, see [`emissions.csv`](emissions.csv)). ### Results Evaluated once on the leakage-free test split of [CIRCL/vulnerability-scores](https://huggingface.co/datasets/CIRCL/vulnerability-scores/tree/55a886a483bb27aabb6461edc64e8058cc136e24) (revision `55a886a`, 70,472 labelled rows): - Accuracy: **0.7415** (95% CI 0.7383 – 0.7447) - F1 Macro: 0.6460 | Class | Precision | Recall | F1 | |-------|----------:|-------:|---:| | Low | 0.6146 | 0.3280 | 0.4278 | | Medium | 0.7767 | 0.8251 | 0.8002 | | High | 0.7328 | 0.7237 | 0.7282 | | Critical | 0.6460 | 0.6107 | 0.6279 | ### Training results Validation metrics at each epoch: | Training Loss | Epoch | Step | Validation Loss | Accuracy | F1 Macro | Low Precision | Low Recall | Low F1 | Medium Precision | Medium Recall | Medium F1 | High Precision | High Recall | High F1 | Critical Precision | Critical Recall | Critical F1 | |:-------------:|:-----:|:-----:|:---------------:|:--------:|:--------:|:-------------:|:----------:|:------:|:----------------:|:-------------:|:---------:|:--------------:|:-----------:|:-------:|:------------------:|:---------------:|:-----------:| | 2.5082 | 1.0 | 17749 | 2.7143 | 0.7229 | 0.6180 | 0.6764 | 0.2567 | 0.3722 | 0.8047 | 0.7647 | 0.7842 | 0.6872 | 0.7424 | 0.7137 | 0.5700 | 0.6374 | 0.6018 | | 2.3903 | 2.0 | 35498 | 2.5364 | 0.7437 | 0.6474 | 0.6961 | 0.3085 | 0.4275 | 0.7932 | 0.8130 | 0.8030 | 0.7202 | 0.7355 | 0.7277 | 0.6314 | 0.6317 | 0.6316 | | 1.8962 | 3.0 | 53247 | 2.6291 | 0.7461 | 0.6624 | 0.5688 | 0.4115 | 0.4775 | 0.7755 | 0.8330 | 0.8032 | 0.7455 | 0.7106 | 0.7276 | 0.6600 | 0.6234 | 0.6412 | | 1.8782 | 4.0 | 70996 | 2.7383 | 0.7500 | 0.6608 | 0.6555 | 0.3622 | 0.4665 | 0.7892 | 0.8279 | 0.8081 | 0.7315 | 0.7352 | 0.7333 | 0.6601 | 0.6120 | 0.6351 | | 1.4131 | 5.0 | 88745 | 3.0105 | 0.7457 | 0.6639 | 0.5635 | 0.4309 | 0.4884 | 0.7916 | 0.8126 | 0.8019 | 0.7271 | 0.7365 | 0.7317 | 0.6589 | 0.6101 | 0.6336 | ### Previous versions The previous model, trained on the earlier random split, is revision [`10d9861`](https://huggingface.co/CIRCL/vulnerability-severity-classification-roberta-base/tree/10d9861) (dataset revision `34e6a14`, test accuracy 81.17%, 73.02% on the test rows without a twin in train). ### Framework versions - Transformers 5.21.0 - Pytorch 2.15.0+cu132 - Datasets 4.8.5 - Tokenizers 0.23.3 ## Citation ```bibtex @article{bonhomme2025vlai, title = {{VLAI}: A {RoBERTa}-Based Model for Automated Vulnerability Severity Classification}, author = {Bonhomme, C{\'e}dric and Dulaunoy, Alexandre}, journal = {arXiv preprint arXiv:2507.03607}, year = {2025} } @techreport{bonhomme2026vulnscores, title = {Improving vulnerability-scores: Leakage-Free Splits and Consistent {CVSS} 4.0 Scoring}, author = {Bonhomme, C{\'e}dric}, institution = {CIRCL - Computer Incident Response Center Luxembourg}, year = {2026}, month = oct, doi = {10.5281/zenodo.23760802}, url = {https://doi.org/10.5281/zenodo.23760802} } ```