Orca-Sonar v5 — 7-class document topic classifier (mmBERT-small) + fp16 ONNX

.gitattributes

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+tokenizer.json filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,130 @@
+---
+license: apache-2.0
+language:
+- de
+- en
+metrics:
+- f1
+- accuracy
+base_model:
+- jhu-clsp/mmBERT-small
+pipeline_tag: text-classification
+tags:
+- document-classification
+- topic-classification
+- security
+- dlp
+- patronus
+- multilingual
+- modernbert
+- text-classification
+- safetensors
+- german
+- english 
+- llm-guard
+- protect-ai
+
+---
+
+# Model Card for Orca-Sonar
+
+**Multilingual Document Topic Classifier for Real-World AI Security & DLP**
+
+Orca-Sonar is a Multilingual ModernBERT-based ([mmBERT](https://huggingface.co/blog/mmbert)) classifier that
+assigns a document/text to one of **7 topic classes**. It is part of the Patronus Protect security stack and
+is designed for topic-/risk-routing of incoming texts (e.g. before they reach an LLM, a DLP gate, or a storage tier).
+
+It classifies German and English text and is robust to **user-to-AI wrappers** (e.g. *"Summarize this contract: …"*),
+i.e. the *topic* of the content determines the class, not the surface format of the request.
+
+## Intended Uses
+
+The model maps an input text to one of:
+
+| id | label | description |
+|---|---|---|
+| 0 | `finance` | invoices, balance sheets, quarterly/annual reports, cash-flow, SEC filings, forecasts |
+| 1 | `hr` | CVs, job ads, employment contracts, terminations, HR policies, performance reviews, recruiting |
+| 2 | `internal_and_tech` | ADRs, RFCs, postmortems, specs, READMEs, wikis, architecture & strategy memos, runbooks |
+| 3 | `legal` | contracts, NDAs, ToS/AGB, privacy policies, statutes/judgments, compliance, legal correspondence |
+| 4 | `marketing` | press releases, newsletters, landing-page/sales copy, outbound pitches, case studies |
+| 5 | `other` | conversational / non-business: smalltalk, recipes, travel, hobby, learning, creative |
+| 6 | `source_code` | raw program code & configs (Python/Go/Rust/JS/TS/SQL/Bash/Dockerfile/k8s/Terraform …) |
+
+**Disambiguation:** on a tie, the more sensitive class wins —
+`legal > hr > finance > internal_and_tech > source_code > marketing > other`.
+
+## Limitations
+
+- Highly accurate on German and English; other languages were not actively tested.
+- The model can produce false positives; for high-stakes routing combine it with a confidence/abstention gate.
+- Robustness against adversarial / out-of-distribution / pure-PII / pathological-length inputs is partial; pair the
+  model with a deterministic pre-gate (length + PII) for production DLP use.
+
+## Model Variants
+
+- **orca-sonar** – full model (`model.safetensors`, fp32).
+- **orca-sonar-fp16 (ONNX)** – FP16 ONNX export under `onnx/onnx_fp16/` — half the size, argmax-faithful to the full model.
+
+# Training Data
+
+Trained on our own in-house dataset (German + English, 7 topic classes), purpose-built for this model.
+**The dataset will be published soon.**
+
+# Benchmark
+
+Held-out test set (**100 % real data**), per-class F1:
+
+| Metric | Score |
+|---|---|
+| **Accuracy** | **0.978** |
+| **F1 (macro)** | **0.978** |
+| F1 legal | 0.995 |
+| F1 source_code | 0.985 |
+| F1 marketing | 0.980 |
+| F1 internal_and_tech | 0.977 |
+| F1 hr | 0.971 |
+| F1 finance | 0.970 |
+| F1 other | 0.970 |
+
+Out-of-distribution **wrapper-robustness probe** (hand-curated, user-to-AI wrapped real content):
+**29/30 (97 %)** — the topic is recognized through the request wrapper across all business classes.
+
+# Usage
+
+```python
+from transformers import pipeline
+
+clf = pipeline("text-classification", model="patronus-studio/orca-sonar-document-classifier")
+clf("Fasse mir diesen Dienstleistungsvertrag zusammen: Laufzeit 24 Monate, Gerichtsstand München …")
+# -> [{'label': 'legal', 'score': 0.99}]
+```
+
+## ONNX
+
+An FP16 ONNX version is available under `onnx/onnx_fp16/`:
+
+```python
+import torch
+from optimum.onnxruntime import ORTModelForSequenceClassification
+from transformers import AutoTokenizer
+
+model_id = "patronus-studio/orca-sonar-document-classifier"
+tokenizer = AutoTokenizer.from_pretrained(model_id)
+model = ORTModelForSequenceClassification.from_pretrained(model_id, subfolder="onnx/onnx_fp16")
+
+inputs = tokenizer("def add(a, b):\n    return a + b", return_tensors="pt")
+logits = model(**inputs).logits
+print(model.config.id2label[int(torch.argmax(logits, dim=-1))])
+```
+
+## Citation
+
+```bibtex
+@misc{orcasonar2026,
+  title={Orca-Sonar: Multilingual Document Topic Classification for Real-World AI Security},
+  author={Patronus Protect},
+  year={2026},
+  howpublished={\url{https://huggingface.co/patronus-studio/orca-sonar-document-classifier}}
+}
+```

config.json

@@ -0,0 +1,98 @@
+{
+  "architectures": [
+    "ModernBertForSequenceClassification"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "bos_token_id": 2,
+  "classifier_activation": "gelu",
+  "classifier_bias": false,
+  "classifier_dropout": 0.0,
+  "classifier_pooling": "mean",
+  "cls_token_id": 1,
+  "decoder_bias": true,
+  "deterministic_flash_attn": false,
+  "dtype": "float32",
+  "embedding_dropout": 0.0,
+  "eos_token_id": 1,
+  "global_attn_every_n_layers": 3,
+  "gradient_checkpointing": false,
+  "hidden_activation": "gelu",
+  "hidden_size": 384,
+  "id2label": {
+    "0": "finance",
+    "1": "hr",
+    "2": "internal_and_tech",
+    "3": "legal",
+    "4": "marketing",
+    "5": "other",
+    "6": "source_code"
+  },
+  "initializer_cutoff_factor": 2.0,
+  "initializer_range": 0.02,
+  "intermediate_size": 1152,
+  "label2id": {
+    "finance": 0,
+    "hr": 1,
+    "internal_and_tech": 2,
+    "legal": 3,
+    "marketing": 4,
+    "other": 5,
+    "source_code": 6
+  },
+  "layer_norm_eps": 1e-05,
+  "layer_types": [
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention"
+  ],
+  "local_attention": 128,
+  "mask_token_id": 4,
+  "max_position_embeddings": 8192,
+  "mlp_bias": false,
+  "mlp_dropout": 0.0,
+  "model_type": "modernbert",
+  "norm_bias": false,
+  "norm_eps": 1e-05,
+  "num_attention_heads": 6,
+  "num_hidden_layers": 22,
+  "pad_token_id": 0,
+  "position_embedding_type": "sans_pos",
+  "problem_type": "single_label_classification",
+  "rope_parameters": {
+    "full_attention": {
+      "rope_theta": 160000,
+      "rope_type": "default"
+    },
+    "sliding_attention": {
+      "rope_theta": 160000,
+      "rope_type": "default"
+    }
+  },
+  "sep_token_id": 1,
+  "sparse_pred_ignore_index": -100,
+  "sparse_prediction": false,
+  "tie_word_embeddings": true,
+  "transformers_version": "5.8.1",
+  "use_cache": false,
+  "vocab_size": 256000
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5c86f42cb39ce056287313b5bec4942bc54d5d2ce1326d2bae702c954d426e52
+size 562591100

onnx/onnx_fp16/model_fp16.onnx

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:888e86b79fcaf8de7e9b0c6461f7ced1e345e7d6fc0388b47e97fbb3401dbfb4
+size 281759682

onnx/onnx_fp16/tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4379fe7f4a7af10b1ac15ffc8ae90f446f92082393699e9dd54b8c3b5ec0abad
+size 34363286

onnx/onnx_fp16/tokenizer_config.json

@@ -0,0 +1,25 @@
+{
+  "backend": "tokenizers",
+  "bos_token": "<bos>",
+  "clean_up_tokenization_spaces": false,
+  "cls_token": "<bos>",
+  "eos_token": "<eos>",
+  "extra_special_tokens": [
+    "<start_of_turn>",
+    "<end_of_turn>"
+  ],
+  "is_local": false,
+  "local_files_only": false,
+  "mask_token": "<mask>",
+  "model_input_names": [
+    "input_ids",
+    "attention_mask"
+  ],
+  "model_max_length": 8192,
+  "pad_token": "<pad>",
+  "padding_side": "right",
+  "sep_token": "<eos>",
+  "spaces_between_special_tokens": false,
+  "tokenizer_class": "TokenizersBackend",
+  "unk_token": "<unk>"
+}

tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4379fe7f4a7af10b1ac15ffc8ae90f446f92082393699e9dd54b8c3b5ec0abad
+size 34363286

tokenizer_config.json

@@ -0,0 +1,25 @@
+{
+  "backend": "tokenizers",
+  "bos_token": "<bos>",
+  "clean_up_tokenization_spaces": false,
+  "cls_token": "<bos>",
+  "eos_token": "<eos>",
+  "extra_special_tokens": [
+    "<start_of_turn>",
+    "<end_of_turn>"
+  ],
+  "is_local": false,
+  "local_files_only": false,
+  "mask_token": "<mask>",
+  "model_input_names": [
+    "input_ids",
+    "attention_mask"
+  ],
+  "model_max_length": 8192,
+  "pad_token": "<pad>",
+  "padding_side": "right",
+  "sep_token": "<eos>",
+  "spaces_between_special_tokens": false,
+  "tokenizer_class": "TokenizersBackend",
+  "unk_token": "<unk>"
+}

training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4c7e34b9f8728ad03118313718aa5221f7524aabc42576b3d5c62809693ca052
+size 5329