Llama-3.2-3B-CVE-Severity-QLoRA

Built with Llama. A QLoRA adapter for meta-llama/Llama-3.2-3B-Instruct that reads a CVE description and predicts its NVD CVSS v3.1 base score (0.0 to 10.0).

Training

• Data: the lite train split of anzil010/cve-severity-guesser: 16,000 CVEs published 2022-03-16 to 2025-09-10, with score hints removed from the descriptions.
• Target: the answer Score: <NVD score>; the loss is computed on the answer only.
• Method: QLoRA. The base model is loaded in 4-bit (NF4, float16 compute) and LoRA layers are trained on the attention projections (q, k, v, o).
• Settings: r 32, alpha 64, dropout 0.1, 1 epoch (1,000 steps of 16 CVEs), learning rate 1e-4 with cosine decay and 3% warm-up, max length 1,024 tokens, seed 42.
• Hardware: one free Google Colab T4 GPU, 131 minutes.
• Validation loss (500 validation CVEs) fell from 0.35 at step 100 to 0.236 at step 1,000, without rising.

Results

Avg error is the mean absolute difference from the NVD score, in points. "Near-copy" test CVEs have a train description with TF-IDF similarity of at least 0.8; "new" ones don't. A dangerous miss is a Critical CVE predicted as Low or Medium. LITE test CVEs were published 2026-03-26 to 2026-10-30; FULL test covers the same period with all 9,939 CVEs.

Model	Test set	Avg error	Near-copy	New	Band accuracy	Dangerous misses
Llama 3.2 3B Instruct, zero-shot	LITE (2,000)	1.709	2.068	1.702	43.7%	26
This adapter	LITE (2,000)	0.897	0.911	0.896	67.3%	26
This adapter	FULL (9,939)	0.922	0.871	0.924	65.2%	96

How to use

import torch
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig

base = "meta-llama/Llama-3.2-3B-Instruct"
tokenizer = AutoTokenizer.from_pretrained(base)
quantization = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type="nf4", bnb_4bit_compute_dtype=torch.float16)
model = AutoModelForCausalLM.from_pretrained(base, quantization_config=quantization, dtype=torch.float16, device_map="auto")
model = PeftModel.from_pretrained(model, "anzil010/Llama-3.2-3B-CVE-Severity-QLoRA")

system = 'You are a vulnerability analyst. Read the CVE description and estimate the CVSS v3.1 base score (0.0 to 10.0) that NVD would assign to it. Return only the score.'
description = "SQL injection in the login form of Example CMS 1.0 allows unauthenticated attackers to run SQL commands."
messages = [{"role": "system", "content": system}, {"role": "user", "content": description}]
prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True) + "Score:"
inputs = tokenizer(prompt, return_tensors="pt", add_special_tokens=False).to(model.device)
output = model.generate(**inputs, max_new_tokens=8, do_sample=False)
print(tokenizer.decode(output[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True))

Limitations

• It learned to score in NVD's style; CNAs often score the same CVE differently.
• Trained on English descriptions published 2022 to 2025; newer kinds of vulnerabilities may score worse.
• A predicted score is a hint for triage, not a replacement for an analyst.

License

Llama 3.2 is licensed under the Llama 3.2 Community License, Copyright © Meta Platforms, Inc. All Rights Reserved. See LICENSE.txt, USE_POLICY.md and NOTICE in this repo. This product uses data from the NVD API but is not endorsed or certified by the NVD.