mailroom-modernbert-classifier

Hierarchical document classifier for the LLM-Mailroom intake pipeline: a fine-tuned ModernBERT-base encoder with a doc_type head plus one subclass head per document class. It is the deterministic pre-check in the BERT-coupled intake overhaul (mailroom-issues #85).

Architecture

• Backbone: answerdotai/ModernBERT-base — 22 layers, 768 hidden, 8,192-token context, bf16.
• Heads: one head per class — doc_type (6 classes) plus 5 subclass heads (contract, corporate_record, correspondence, insurance_claim, merger_agreement). MLP heads with dropout 0.1.
• Windowing: token-level, 8,192 tokens with 512-token overlap. doc_type by plurality vote over windows; subclass by plurality over windows whose doc_type vote is the winning class.
• Calibration: per-head temperature scaling (temperatures.json).

Files

file	purpose
model.safetensors	ModernBERT backbone weights (bf16, ~298 MB)
heads.pt	hierarchical head state dicts
labels.json	head vocabularies (labels / label2id / id2label / weights)
temperatures.json	per-head calibration temperatures
train_counts.json	per-(doc_type, subclass) authentic train-row counts (support gate)
config.json, tokenizer.json, tokenizer_config.json	backbone config + tokenizer
summary.json	full run summary (hyperparameters, per-epoch metrics, selection, test metrics)

Labels

doc_type (6): contract, merger_agreement, corporate_record, correspondence, insurance_claim, unknown (inference-only abstention — not a trained class)

contract (24): agency, co_branding, collaboration, consulting, development, distributor, endorsement, franchise, hosting, ip, joint_venture, license, maintenance, manufacturing, marketing, other, outsourcing, promotion, reseller, service, sponsorship, strategic_alliance, supply, transportation

corporate_record (10): articles_of_incorporation, board_resolution, bylaws, charter_amendment, indenture, officer_certificate, other, powers_of_attorney, rights_instrument, subsidiary_list

correspondence (7): demand, email, letter, meeting_request, memo, notice, press_release

insurance_claim (6): auto, carrier, inpatient, outpatient, pde, property

merger_agreement (5): all_cash, all_stock, mixed_cash_stock, mixed_cash_stock_election, other

Results

Trained 2 epochs on Lucius-Morningstar/mailroom-modernbert-training @ 5b72a345cd3c057b736bea4910fdbef6509ad1c3 — 4,497 train / 489 validation windows; 323 held-out test documents (never used for training, calibration, or threshold tuning).

Validation

epoch	val_loss	doc_acc	macro-F1 (observed)	ECE (calibrated)
1	1.0041	0.8859	0.8524	0.0201
2	0.9121	0.9195	0.9051	0.0205

Selected epoch: 2 — best observed doc_type macro-F1 subject to calibrated ECE ≤ 0.05 (gate_met: true).

Held-out test (323 docs)

metric	value
doc_type accuracy	0.9319 (301/323)
subclass accuracy (given correct doc_type)	0.5449 (164/301)

Per-head (epoch 2, validation)

head	window acc	macro-F1 (observed)	ECE (calibrated)
doc_type	0.9141	0.9051	0.0205
insurance_claim	0.8889	0.8085	0.0500
corporate_record	0.5405	0.2218	0.1037
merger_agreement	0.4656	0.1988	0.0350
correspondence	0.5161	0.0980	0.1050
contract	0.1630	0.0899	0.0630

Usage

The backbone is a standard ModernBertModel; the hierarchical heads are a custom bundle. Load it with the mailroom-ml inference layer:

from mailroom_ml.inference import load_bundle, classify_document

bundle = load_bundle("Lucius-Morningstar/mailroom-modernbert-classifier")
result = classify_document(
    title="Notice of Default",
    text=document_text,
    bundle=bundle,
)
# result["doc_type"], result["subclass"], result["confidence"], result["route"]

route == "fast_path" means the calibrated gate passed and the LLM sorter may be skipped (skip mode + allowlist only); otherwise route the document to the LLM sorter with the BERT triage as an advisory prior.

Provenance

• Training run: 20260920-173810 — 2 epochs, ~3.78 h on a Modal L4.
• Hyperparameters: batch 4, grad-accum 8, lr 2e-5, seed 42, λ_dt 0.65, label smoothing 0.05, sqrt-inverse class weights (cap 10), MLP heads (dropout 0.1), weight decay 0.01, subclass support floor 12.
• Dataset revision: 5b72a345cd3c057b736bea4910fdbef6509ad1c3.
• Calibration temperatures: doc_type 0.484, contract 0.705, corporate_record 0.596, correspondence 0.960, insurance_claim 0.122, merger_agreement 0.679.

Limitations

• Subclass heads are weak for contract / correspondence / corporate_record / merger_agreement (macro-F1 0.09–0.22). Use the doc_type head for routing; route subclass-ambiguous documents to the LLM sorter (the Tier-1 prior-scoped lane, mailroom-issues #108). Do not trust a skip-mode subclass for these classes yet.
• Trained on a curated legal-document corpus; not a substitute for legal review.
• unknown is an inference-only abstention label, not a trained class.
• Long documents are windowed (8,192 tokens, 512 overlap); the model never truncates silently — oversize documents fall back to the LLM path.

License

Apache-2.0 (inherits answerdotai/ModernBERT-base).

Revision — run-3 (2026-09-21)

Checkpoint runs/20260921-132753 (run_id 20260921-093211; this repo's current head).

Config: epochs=2 · batch=4 · grad-accum=8 · lr=2e-5 · seed=42 · max_length=8192 · label-smoothing=0.05 · loss-lambda-dt=0.65 · weight-mode=sqrt-inverse · mlp-heads.

Held-out test (323 docs @ 8,192, L4; reports/eval_run3_20260921.json / eval_report_run3.json)	run-2	run-3
doc_type accuracy	0.799	0.8947
window calibration ECE	—	0.0203 (504 windows)
subclass accuracy (conditional)	—	0.526
selective-risk threshold (2% budget, min-n≥30, Wilson)	—	0.93 (coverage 43.3%, budget_met)

Validation selection: epoch 1 — doc_type macro-F1 0.9245, calibrated ECE 0.0228 (gate_met true; the run-2 artifact shipped gate_met false). Subclass heads: insurance_claim 0.7714 → 0.8954 (e2) · corporate_record 0.238 → 0.2669 · merger 0.195 → 0.218; contract/correspondence unchanged (0.039 / 0.085) — excluded from the fast path by the shipped per-head ECE policy (see summary.json → checkpoint_selection.head_exclusion_policy; contract 0.044 + doc_type 0.023 pass, the other four heads excluded).

Serving: ONNX fp32 export parity PASS (≤ 4.3e-6 per head vs 1e-4 gate); the int8 dynamic variant is decision-unsafe (argmax agreement 0.8/0.2/0.6 on doc_type/contract/corporate) — runtime serves fp32 by default. Bundled-tokenizer padding/truncation neutralized at load (serving fix, mailroom-ml 06842ba); CPU p50: 75 ms @64 tok / 1.63 s @1,024 tok (fp32, Apple M2).