Unify Vela model cards with Semantic Router branding

README.md

@@ -19,14 +19,22 @@ tags:
 - long-context
 ---
 
-# Vela 1.0 · FactCheck
+<div align="center">
+  <img src="https://vllm-sr.ai/img/vllm-sr-logo.social.png" alt="vLLM Semantic Router" width="560" />
+  <p>
+    <a href="https://vllm-sr.ai/"><strong>Docs</strong></a> |
+    <a href="https://vllm-sr.ai/blog/"><strong>Blog</strong></a> |
+    <a href="https://vllm-dev.slack.com/archives/C09CTGF8KCN"><strong>Slack</strong></a> |
+    <a href="https://github.com/vllm-project/semantic-router"><strong>GitHub</strong></a>
+  </p>
+</div>
+
+# Vela-1.0-Encoder-307M-FactCheck
 
 **Know when to bring in factual verification.**
 
 307M · Multilingual · 32K context
 
-[Collection](https://huggingface.co/collections/llm-semantic-router/vela-10-router-models-6aa555ba70cc6997d6d67798) · [vLLM Semantic Router](https://github.com/vllm-project/semantic-router)
-
 Vela FactCheck identifies requests that need factual verification. It helps a router decide when to add retrieval, external evidence, or a verification step before answering.
 
 ## What it brings
@@ -42,7 +50,7 @@ Vela FactCheck identifies requests that need factual verification. It helps a ro
 | Verification recall | 30% | **90%** |
 | Verification precision | 75% | **75%** |
 
-Higher recall comes with more verification calls. See the evaluation for operating-point guidance.
+Higher recall comes with more verification calls; choose the operating threshold for your traffic.
 
 ## Quick start
 
@@ -66,12 +74,9 @@ Choose the signals your router needs. Every model has a focused role.
 | Role | Models |
 |---|---|
 | Understand requests | [Domain](https://huggingface.co/llm-semantic-router/Vela-1.0-Encoder-307M-Domain) · [Feedback](https://huggingface.co/llm-semantic-router/Vela-1.0-Encoder-307M-Feedback) · [Modality](https://huggingface.co/llm-semantic-router/Vela-1.0-Encoder-307M-Modality) |
-| Detect risks | PromptGuard¹ · [PII](https://huggingface.co/llm-semantic-router/Vela-1.0-Encoder-307M-PII) · Safety¹ · Hazard¹ |
+| Detect risks | Guard¹ · [PII](https://huggingface.co/llm-semantic-router/Vela-1.0-Encoder-307M-PII) · Safety¹ · Hazard¹ |
 | Decide when to verify | [FactCheck](https://huggingface.co/llm-semantic-router/Vela-1.0-Encoder-307M-FactCheck) |
 | Retrieve context | [Embedding](https://huggingface.co/llm-semantic-router/Vela-1.0-Encoder-307M-Embedding) · [Reranker](https://huggingface.co/llm-semantic-router/Vela-1.0-Encoder-307M-Reranker) |
 | Build new capabilities | [Encoder](https://huggingface.co/llm-semantic-router/Vela-1.0-Encoder-307M) |
 
 ¹ Coming soon. Explore available models in the [Vela collection](https://huggingface.co/collections/llm-semantic-router/vela-10-router-models-6aa555ba70cc6997d6d67798).
-
-
-[Documentation and evaluation](TECHNICAL.md)

TECHNICAL.md

@@ -1,184 +0,0 @@
----
-license: mit
-library_name: transformers
-pipeline_tag: text-classification
-base_model: llm-semantic-router/Vela-1.0-Encoder-307M
-language:
-- en
-- zh
-- es
-- fr
-- de
-- ja
-tags:
-- vela
-- modernbert
-- router
-- fact-checking
-- multilingual
-- long-context
----
-
-# Vela-1.0-Encoder-307M-FactCheck
-
-This model predicts whether a **request needs external factual verification**. It does not judge whether a generated answer is true, retrieve evidence, or verify a claim itself. A request to extract information fully supplied in a passage, transform wording, calculate a self-contained result, or write fiction can be `NO_FACT_CHECK_NEEDED`; questions requiring external factual knowledge are `FACT_CHECK_NEEDED`.
-
-On a frozen set of 102 natural requests, native Transformers FP32 accuracy improved from **63/102 to 79/102**, and macro F1 improved from **0.579 to 0.772**. These are small, source-group-disjoint task tests with assistant-reviewed labels, not expert human annotations or a general-purpose fact-checking benchmark. More requests needing verification are detected, with a substantial increase in unnecessary-verification false positives. Detailed results and limitations follow.
-
-## Architecture and lineage
-
-| Property | Value |
-|---|---|
-| Architecture | Standard `ModernBertForSequenceClassification` |
-| Encoder | 22 layers, width 768, 12 attention heads |
-| Shared encoder parameters | 306,939,648 |
-| Total unique parameters, including classification head | 307,531,778 |
-| Pooling | Masked mean pooling |
-| Context capacity | 32,768 tokens, including special tokens |
-| Published weights | FP32 |
-| Labels | `0=NO_FACT_CHECK_NEEDED`, `1=FACT_CHECK_NEEDED` |
-| Weight SHA-256 | `809d11b1b44025d5c985859ca2f10c720c1afef88024d6d21ecb0aa25cbe49a3` |
-
-“307M” names the shared encoder size. This is a dense encoder with a two-class head, not a generative router or an MoE model. Router-compatible `fact_check_mapping.json` and the HF `config.json` use the same label indices; mappings and weights were hashed together before final testing.
-
-The final base is [Vela-1.0-Encoder-307M at revision 5fe5bbb1a88b7fdcc93bb5b9d546c574564eb114](https://huggingface.co/llm-semantic-router/Vela-1.0-Encoder-307M/tree/5fe5bbb1a88b7fdcc93bb5b9d546c574564eb114), with base weight SHA-256 `ccc274d12ef6f94f54c0c23f0388527b0a925268a1332b3a6e5c8e56868efb00`.
-
-Training started with a fresh task head and rank-32 LoRA adapter on [mmbert-32k-yarn at revision 72a23a6640489471eb4ff7ad3ec5bc80af8a27de](https://huggingface.co/llm-semantic-router/mmbert-32k-yarn/tree/72a23a6640489471eb4ff7ad3ec5bc80af8a27de), followed by natural-request and long-context training. The selected adapter and task head were then transferred to the published Vela Base. Direct transfer failed the development 32K retention gate; bounded adaptation on existing training rows restored development retention. The final adapter was chosen at step 100 of that 200-step adaptation and merged into the actual Vela Base. Original FactCheck task weights were used only as the baseline, not as an initialization.
-
-## Independent final results
-
-The candidate and data hashes were frozen before the first final prediction. The baseline is [mmbert32k-factcheck-classifier-merged at revision 290d3182b2c87e489bd4f63c889e803a27599681](https://huggingface.co/llm-semantic-router/mmbert32k-factcheck-classifier-merged/tree/290d3182b2c87e489bd4f63c889e803a27599681). Baseline and candidate use identical text, tokenizer budgets, standard HF execution, and argmax over two logits. This corresponds to a 0.5 positive-score boundary; the reviewed Router canonical default is 0.6, the checked-in configuration example explicitly uses 0.65, and 0.7 is only the classifier's fallback when supplied a nonpositive threshold. These are different operating points. The primary table uses **FP32 weights and FP32 forward computation**. No weight was changed after final testing. The separate threshold procedure below used only development scores and was locked before its fixed-threshold final analysis.
-
-| Final subset | Requests | Baseline correct | Vela correct | Baseline macro F1 | Vela macro F1 |
-|---|---:|---:|---:|---:|---:|
-| Natural requests, all sources | 102 | 63 | 79 | 0.579 | 0.772 |
-| Aya original reviewed subset | 18 | 5 | 15 | 0.257 | 0.734 |
-| Additional Aya reviewed subset | 29 | 19 | 22 | 0.651 | 0.709 |
-| Dolly reviewed subset, preserving supplied context | 55 | 39 | 42 | 0.633 | 0.752 |
-| Weak factual-question / authored-transformation supervision | 720 | 448 | 719 | 0.572 | 0.999 |
-| Authored long-context stress | 48 | 24 | 35 | 0.333 | 0.708 |
-
-The 720 weak or authored examples are deliberately separate from natural requests. The mixed 870-row score is dominated by these easier examples and must not be reported as natural-request accuracy. Natural labels were individually reviewed by an assistant before model predictions. Contributor/source groups stay within one split; older-model and base-pretraining exposure cannot be ruled out. The baseline's historical recipe used some Dolly instructions without a complete row-level manifest.
-
-At the fixed argmax operating point, among 50 natural requests needing verification, recall increased from **16/50 to 45/50**. Among 52 natural requests not needing verification, false positives increased from **5/52 to 18/52**. This trades fewer missed checks for more unnecessary checks and their associated latency or cost. Softmax scores are not calibrated probabilities. A deployment threshold requires its own development-based selection and evaluation.
-
-Natural German results regressed from **4/6 to 2/6** correct. The natural final sample includes English, German, Spanish, and French; multilingual training and authored Chinese/Japanese examples do not establish equal natural-request quality in all six training languages. The included source-group bootstrap describes uncertainty; for the argmax comparison, the natural accuracy-improvement interval crosses zero despite the positive point estimate.
-
-| Attended input tokens | Authored final cases | Baseline correct | Vela correct | Vela macro F1 |
-|---|---:|---:|---:|---:|
-| 4,096 | 12 | 6 | 9 | 0.733 |
-| 8,192 | 12 | 6 | 9 | 0.733 |
-| 16,384 | 12 | 6 | 9 | 0.733 |
-| 32,768 | 12 | 6 | 8 | 0.625 |
-
-These inputs contain repeated original background narratives with held-out source payloads at different positions. They are controlled stress, not naturally distributed long documents; length/position variants sharing a payload are correlated. The 32K final result is **8/12**, below development's 12/12. Long-context acceptance and finite logits do not establish reliable long-document decisions. All 24 positive long cases were detected, but 13 of 24 negative long cases were falsely flagged.
-
-A separate BF16 audit of the same immutable weights and final rows gave 80/102 natural correct. Only 1/870 candidate predictions differed from FP32: a French natural negative that FP32 misclassified. All long-case predictions agreed. Baseline BF16/FP32 differed on four weak-supervision rows and agreed on natural and long rows. Neither precision comparison was used to retune the candidate.
-
-## Router operating point
-
-A documentation correction distinguishes the existing Router settings: `defaultHallucinationModule` in `src/semantic-router/pkg/config/canonical_defaults.go` supplies **0.6**, `config/config.yaml` explicitly sets **0.65** in its example, and `FactCheckClassifier.Classify` uses **0.7** only when its configured threshold is nonpositive. The initial card incorrectly called 0.7 the Router default. The historical selection-plan/lock field retains that original descriptive error for audit; it did not alter the fixed selection grid or development objective. [Correction evidence](evaluation/router-default-correction.json) and [all fixed operating points](evaluation/operating-points-fp32-defaults-correction.json) supersede that attribution. No weights, predictions, or selected threshold changed.
-
-The suggested Vela Router threshold is **0.85**. This was selected before evaluating its final operating point, from a prewritten grid of 0.50–0.95 in increments of 0.025. The development objective gave equal weight to three natural-request sources and the long-context source; weak supervision did not dominate selection. Exact ties chose the larger threshold. The objective was 0.87905 at 0.5, 0.88317 at 0.7, and 0.88725 at 0.85. The [selection plan](evaluation/threshold-selection-plan.json) and [threshold lock](evaluation/threshold-lock.json) precede the fixed-threshold final analysis.
-
-For the 102 natural final requests, the confusion entries below use 52 negative and 50 positive gold requests. “Checks” counts requests the binary policy would send for verification; it does not measure actual remote calls or their latency.
-
-| Model / threshold | TN | FP | FN | TP | Precision | Recall | Checks | Macro F1 |
-|---|---:|---:|---:|---:|---:|---:|---:|---:|
-| Baseline / 0.5 argmax | 47 | 5 | 34 | 16 | 0.762 | 0.320 | 21 | 0.579 |
-| Baseline / 0.6 canonical default | 47 | 5 | 35 | 15 | 0.750 | 0.300 | 20 | 0.565 |
-| Baseline / 0.65 configuration example | 48 | 4 | 35 | 15 | 0.789 | 0.300 | 19 | 0.573 |
-| Baseline / 0.7 nonpositive-argument fallback | 49 | 3 | 36 | 14 | 0.824 | 0.280 | 17 | 0.567 |
-| Baseline / 0.85 | 51 | 1 | 37 | 13 | 0.929 | 0.260 | 14 | 0.567 |
-| Vela / 0.5 argmax | 34 | 18 | 5 | 45 | 0.714 | 0.900 | 63 | 0.772 |
-| Vela / 0.6 | 35 | 17 | 5 | 45 | 0.726 | 0.900 | 62 | 0.782 |
-| Vela / 0.65 | 35 | 17 | 5 | 45 | 0.726 | 0.900 | 62 | 0.782 |
-| Vela / 0.7 | 35 | 17 | 5 | 45 | 0.726 | 0.900 | 62 | 0.782 |
-| Vela / 0.85 development-selected | 37 | 15 | 5 | 45 | 0.750 | 0.900 | 60 | 0.803 |
-
-Relative to the old canonical 0.6 threshold, positive recall increases from **30% to 90%** while positive precision remains **75%** on these 102 requests. At the suggested threshold, natural correctness is **82/102**, but the model still makes more unnecessary-verification decisions than the old canonical default: 15 versus 5. Estimated verification decisions increase from 20/102 to 60/102, primarily because the old model misses most positive requests. This is not a claim of reduced end-to-end cost or latency. German remains a regression, 3/6 correct at 0.85 versus the baseline canonical default's 4/6.
-
-At 0.85, long-stress correctness is 37/48 with macro F1 0.758, positive recall 24/24, and 11 false positives among 24 negatives. The 4K/8K/16K/32K correct counts are 9/12, 10/12, 10/12, and **8/12**. The threshold does not fix the 32K limitation. All fixed-threshold [FP32 results](evaluation/operating-points-fp32.json) and [natural language breakdowns](evaluation/operating-language-fp32.json) are retained.
-
-## Engine qualification
-
-The frozen weights also passed scoped native-engine checks. These check numerical behavior and lifecycle handling; they do not expand the quality benchmark above to every device or engine.
-
-| Qualified path | Cases | Maximum probability difference from reference | Decisions |
-|---|---:|---:|---|
-| Candle CPU | 8 inputs, 9–512 tokens | 3.05e-5 | All argmax labels agree |
-| ONNX Runtime ROCm with project CK attention integration | 12 inputs including 4K/8K/16K/32K | 0.000975 | All argmax labels agree |
-| ONNX Runtime ROCm CK batch/padding profile | 20 B1/B2 cases at 2, 63, 64, 65, 127, 128, 129, 512, 4096, 32768 tokens | 1.64e-5 | All argmax labels agree |
-
-Over-budget input is explicitly rejected, owner close/reopen succeeds, and mismatched label ordering is rejected. The batch/padding profile observed 440 CK attention calls, 19,120 ROCm-provider node executions, and 60 CPU shape-node executions across its cases. This is actual provider-profile evidence, not an assumption from installed GPU providers. The [Candle](evaluation/engines/vela-factcheck-candle-cpu-v1.json), [ONNX](evaluation/engines/vela-factcheck-fp32-ck-shared-onnx-binding-v1.json), and [profile](evaluation/engines/vela-factcheck-fp32-ck-shared-profile-v1.json) reports retain input/reference hashes. Their individual forward times include differing warmup states and are not a serving latency benchmark.
-
-## Training and data
-
-The recorded stages used manual mean cross-entropy with explicit gradient-accumulation normalization, FP32 trainable parameters, BF16 autocast, SDPA, non-reentrant gradient checkpointing, and finite-gradient checks. LoRA rank was 32, alpha 64, dropout 0.05; attention/MLP adapters and the standard task head/classifier were trained.
-
-| Stage | Train / dev rows | Maximum tokens | Steps run / selected | Learning rate | Development selection |
-|---|---:|---:|---:|---:|---|
-| Initial task training | 3,579 / 504 | 2,048 | 600 / 400 | 3e-5 | Macro F1 |
-| Natural-request extension | 3,640 / 593 | 2,048 | 600 / 300 | 2e-5 | Equal-source macro F1 |
-| Long-context curriculum | 3,784 / 641 | 32,768 | 400 / 400 | 8e-6 | Equal-source macro F1 |
-| Vela Base adaptation | 3,676 / 641 | 32,768 | 200 / 100 | 2e-6 | Equal-length macro F1 |
-
-The final adaptation reuses the short training rows and 36 existing 32K training variants; it introduces no development or test examples. Context training, development, and final payload groups and narratives are separate. All gold final cases remain in the results, including regressions.
-
-Sources are [Aya](https://huggingface.co/datasets/CohereLabs/aya_dataset/tree/f9ea04583f02a8f86404ff6c58bf75fe637df8a2) and [Global-MMLU](https://huggingface.co/datasets/CohereLabs/Global-MMLU/tree/0e619dbeb34206cd48705a1a0ea7fb21cae09993), both Apache-2.0, [Dolly](https://huggingface.co/datasets/databricks/databricks-dolly-15k/tree/bdd27f4d94b9c1f951818a7da7fd7aeea5dbff1a), CC-BY-SA-3.0, and original transformation/context material. Dolly attribution: Copyright 2023 Databricks, Inc.; Wikipedia editors and contributors for source passages. Reconstructed text and derived annotations retain their applicable source attribution and terms. The [source manifests and text-free annotations](reproducibility/src/training/model_classifier/sequence_repair/data/README.md) record immutable revisions and hashes. Benchmark-question labels and lexical training rules are explicitly identified as weak task supervision.
-
-## Use
-
-```python
-import torch
-from transformers import AutoModelForSequenceClassification, AutoTokenizer
-
-model_id = "llm-semantic-router/Vela-1.0-Encoder-307M-FactCheck"
-tokenizer = AutoTokenizer.from_pretrained(model_id)
-model = AutoModelForSequenceClassification.from_pretrained(
-    model_id, torch_dtype=torch.float32,
-    attn_implementation="sdpa", reference_compile=False,
-).eval()
-inputs = tokenizer("Translate this sentence without checking its claims.",
-                   return_tensors="pt", truncation=False)
-if inputs["input_ids"].shape[1] > model.config.max_position_embeddings:
-    raise ValueError("Input exceeds the model's explicit context budget")
-with torch.inference_mode():
-    probabilities = model(**inputs).logits.float().softmax(-1)[0]
-label = model.config.id2label[int(probabilities.argmax())]
-```
-
-Budget includes special tokens. Do not infer a 32K quality guarantee or an appropriate default resource budget from the configuration. Runtime token sampling, thresholds, batching, and inference engine can change request-level behavior. Task-quality results above use native HF models; scoped engine qualification is reported separately. GPU forward timings under concurrent training are not HTTP latency.
-
-See [reproduction instructions](reproducibility/README.md), [frozen candidate hashes](evaluation/candidate-lock.json), [FP32 final metrics](evaluation/final-candidate-fp32.json), and [individual predictions](evaluation/final-candidate-fp32.predictions.jsonl). This candidate is intentionally not advertised as improving every language, precision, or long-input condition.
-
-## ONNX artifacts
-
-`onnx/model.onnx` contains the portable FP32 graph. The AMD accelerated
-`onnx/model_fa.onnx` retains FP32 encoder and task weights and uses FP16 only at
-the CK attention boundaries. It requires the project's ROCm CK custom operator.
-Both graphs share `onnx/model.onnx.data`, saving 1.23 GB of duplicate storage.
-
-The current release removes the earlier FP16-weight variants: narrowing the
-encoder also rounded position-frequency buffers, unlike native HF loading.
-Correcting those buffers exposed a short-input FP16 error above the original
-probability tolerance. The retained FP32/CK path passes that unchanged tolerance
-on all 12 binding cases and 20 padded graph cases through 32K. Earlier evidence
-remains historical; the table above links the currently shipped graph tests.
-This is an inference artifact correction; native task weights and the selected
-verification threshold are unchanged. See [export reproduction](reproducibility/onnx/README.md)
-and [exact shared storage](evaluation/engines/factcheck-shared-storage-reproduction.json).
-
-## Frozen LoRA variant
-
-The `lora/` directory contains the exact development-selected adapter, its pinned
-Vela Base parent, final task configuration, tokenizer and a self-contained loader.
-Use `lora/load_adapter.py` to restore configuration before applying the adapter.
-The adapter restores the same native task weights and development-selected
-threshold 0.85. Engine variants are described above.
-
-Three independently loaded, padded bilingual/Unicode CPU batches matched the
-frozen merged checkpoint at the original probability tolerance, with identical
-argmax labels. See [standalone verification](standalone-verification.json) and
-[adapter identity](adapter-qualification.json). This is numerical verification
-of the same task model, not a new accuracy evaluation or a 32K adapter benchmark.