---
library_name: transformers
tags:
- generated_from_trainer
- cybersecurity
- continual-pretraining
- CPT
- text-generation
- casual-lm
- risys-lab
model-index:
- name: Qwen/Qwen3-8B-Base
  results: []
language:
- en
base_model:
- Qwen/Qwen3-8B-Base
pipeline_tag: text-generation
---

# RedSage-Qwen3-8B-CFW

<div align="center">
  <img src="https://img.shields.io/badge/Task-Cybersecurity-red" alt="Cybersecurity"> 
  <img src="https://img.shields.io/badge/Stage-Continual_Pretraining-blue" alt="CPT">
</div>



## Model Summary

**RedSage-Qwen3-8B-CFW** is a cybersecurity-specialized Large Language Model (LLM) developed by [RISys-Lab]. It is the result of **Continued Pre-training (CPT)** on the **CyberFineWeb** corpus.

This model serves as the foundational stage of the RedSage pipeline. It takes the general-purpose [Qwen3-8B-Base](https://huggingface.co/Qwen/Qwen3-8B-Base) and adapts it to the cybersecurity domain using ~11.7 billion tokens of filtered, high-quality cybersecurity web data. To maintain general reasoning capabilities, it utilizes a data replay strategy with educational content.

- **Paper:** [RedSage: A Cybersecurity Generalist LLM](https://openreview.net/forum?id=W4FAenIrQ2) ([arXiv](https://arxiv.org/abs/2601.22159))
- **Repository:** [GitHub](https://github.com/RISys-Lab/RedSage)
- **Base Model:** [Qwen/Qwen3-8B-Base](https://huggingface.co/Qwen/Qwen3-8B-Base)
- **Variant:** CFW (CyberFineWeb Continued Pre-training)

## Intended Use

This model is a **base model** intended for:
1.  Further fine-tuning on downstream cybersecurity tasks.
2.  Research into domain adaptation and continual pre-training dynamics.
3.  Cybersecurity text completion and generation.

**Note:** As a base model, this checkpoint has **not** been instruction-tuned (SFT) or aligned (DPO). It behaves like a completion engine. For a chat-ready assistant, please see `RISys-Lab/RedSage-Qwen3-8B-DPO`.

## Training Lineage

RedSage employs a multi-stage training pipeline. This model represents the output of **Stage 1**.

1.  **Stage 1: Continual Pre-Training (CPT)** -> **`RedSage-Qwen3-8B-CFW`** (Current Model)
    * *Data:* CyberFineWeb (11.8B tokens)
2.  Stage 2: Targeted Pre-Training -> [RedSage-Qwen3-8B-Base](https://huggingface.co/RISys-Lab/RedSage-Qwen3-8B-Base)
4.  Stage 3: Supervised Fine-Tuning (SFT) -> [RedSage-Qwen3-8B-Ins](https://huggingface.co/RISys-Lab/RedSage-Qwen3-8B-Ins)
5.  Stage 4: Direct Preference Optimization (DPO) -> [RedSage-Qwen3-8B-DPO](https://huggingface.co/RISys-Lab/RedSage-Qwen3-8B-DPO)

## Training Data: CyberFineWeb

This model was trained on **CyberFineWeb**, a large-scale cybersecurity corpus constructed by filtering the FineWeb dataset (2013–2024).

1.  **Filtering:** A ModernBERT-base classifier was trained on the Cybersecurity Topic Classification dataset to identify cybersecurity content within Common Crawl.
2.  **Dataset Size:** The filtering process yielded \~125M documents (\~89.8B tokens). We select the latest subset of **~11.7B tokens** for this training stage.
3.  **General Knowledge Replay:** To prevent catastrophic forgetting, we mixed the cybersecurity data with a 30% replay ratio of **FineWeb-Edu** samples.

## Performance

RedSage-8B-CFW demonstrates improved performance over the general-purpose Qwen3-8B-Base on cybersecurity benchmarks while maintaining general capabilities.

### RedSage-Bench (0-shot Accuracy)

| Category | Qwen3-8B-Base | **RedSage-8B-CFW** |
| :--- | :---: | :---: |
| **Macro Average** | 84.24 | **84.86** |
| Knowledge (Gen) | 83.08 | **83.62** |
| Knowledge (Frameworks) | 81.94 | **83.30** |
| Skill (Offensive) | 88.23 | **88.81** |
| Tools (CLI) | 85.08 | **85.30** |
| Tools (Kali) | 78.86 | **79.32** |

### External Cybersecurity Benchmarks (5-shot)

| Benchmark | Qwen3-8B-Base | **RedSage-8B-CFW** |
| :--- | :---: | :---: |
| **Mean** | 80.81 | **82.66** |
| CTI-Bench (MCQ) | **68.80** | 68.40 |
| CTI-Bench (RCM) | 63.50 | **67.60** |
| CyberMetric (500) | 92.00 | **93.80** |
| MMLU (Security) | 83.00 | **86.00** |
| SecBench (En) | 82.84 | **83.62** |
| SecEva (MCQ) | 75.60 | **76.10** |
| SECURE (CWET) | 92.70 | **93.33** |
| SECURE (KCV) | 75.05 | **81.34** |
| SECURE (MEAT) | **93.81** | 93.72 |

## Training Procedure

The model was trained using the [Axolotl](https://github.com/axolotl-ai-cloud/axolotl) framework 

- **Learning Rate:** 2.5e-6 (constant with linear warmup)
- **Optimizer:** AdamW
- **Epochs:** 1

## Usage

```python
from transformers import AutoTokenizer, AutoModelForCausalLM

model_id = "RISys-Lab/RedSage-Qwen3-8B-CFW"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, device_map="auto")

text = "The primary difference between a firewall and an IDS is"
inputs = tokenizer(text, return_tensors="pt").to("cuda")

outputs = model.generate(**inputs, max_new_tokens=50)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
```

## Citation

If you use this model or dataset, please cite our paper:

```
@inproceedings{suryanto2026redsage,
  title={RedSage: A Cybersecurity Generalist {LLM}},
  author={Naufal Suryanto and Muzammal Naseer and Pengfei Li and Syed Talal Wasim and Jinhui Yi and Juergen Gall and Paolo Ceravolo and Ernesto Damiani},
  booktitle={The Fourteenth International Conference on Learning Representations},
  year={2026},
  url={https://openreview.net/forum?id=W4FAenIrQ2}
}
```