NeroT-86M-Exp

An experimental 85.6M-parameter encoder-decoder model for turning a conversation's first user message into a short title.

Quick start

This checkpoint uses a custom MLX architecture and is not a drop-in AutoModel checkpoint. Download the complete repository so the implementation, custom kernels, configuration, tokenizer, and weights remain together:

pip install mlx huggingface_hub tokenizers

from huggingface_hub import snapshot_download

model_dir = snapshot_download("j0no12/NeroT-86M-Exp")
print(model_dir)

Use modeling_titan_mlx.py and kernels.py from the downloaded snapshot. The training-time decode configuration was greedy generation with a minimum length of 3, no repeated token unigrams, and a maximum of 32 tokens. The repository does not yet include a packaged generation CLI, so loading and decoding require custom integration.

Results

TitleBench (full 21,448-example split)

Metric	Value
ROUGE-1	0.152
ROUGE-2	0.025
ROUGE-L	0.151
chrF	0.082
BLEU-4	0.284

Exact match and repetition rate were not reported for this run. The evaluation used the repository's official TitleBench scorer with greedy decoding, min_len=3, unigram repetition blocking, and a 32-token maximum.

Takeaway: This release documents an architecture experiment, not a state-of-the-art checkpoint. Its title-generation quality trails NeroT-29M on the directly comparable overlap metrics.

Architecture

Component	Configuration
Parameters	~85.6M with tied embeddings
Model type	Bidirectional encoder + autoregressive decoder
Encoder / decoder layers	20 / 6
Hidden size	512
Attention	8 query heads, 2 key/value heads, head size 64
FFN	SwiGLU, size 1,408
Position encoding	RoPE, base 10,000
Normalization	RMSNorm, epsilon 1e-6
Vocabulary	16,385 tokens
Maximum source / target lengths	512 / 48
Embeddings	Tied input/output embeddings
Dropout	0.0

Training story

1. Interleaved pre-training

• MASS-style span corruption over lmsys/lmsys-chat-1m conversations.
• Approximately 15% masking with one dedicated mask token.
• Chat-title pairs were upweighted 2× and represented about 45% of batches.

2. Title-only fine-tuning

• Title pairs only.
• Peak learning rate 8e-5 with cosine decay.
• Data processing and deduplication used seed 42.

Data lineage

Source	Role	Recorded volume
j0no12/chat-titles-unified	Real title pairs	223,853 train pairs
Reconstructed BananaMind-Chat-Title-200K	Real/teacher title pairs reconstructed by gated LMSYS ordinal ID	127,335 train pairs
BananaMind/BananaMind-Title-1.0 over fresh LMSYS first messages	Self-distilled titles	248,132 prompts
lmsys/lmsys-chat-1m	Span-corruption corpus	~1M conversations

All 21,448 normalized TitleBench inputs, including locally reconstructed gated rows, were excluded before encoding. The pipeline also performed exact-pair deduplication and input-key priority deduplication with real examples preferred over synthetic examples.

Evaluation details

• Dataset: j0no12/TitleBench, all 21,448 examples.
• Decode: greedy, minimum length 3, no repeated unigrams, maximum 32 tokens.
• Scorer: TitleBench's published scorer.
• Missing metrics: exact match and repetition were not recorded in the original result table and are intentionally not inferred here.

Reproducibility

Item	Value
Model repository revision documented by this card update	74667d68b8edb28729674b1c0f4b85256293b63e
Data-processing seed	42
Implementation	modeling_titan_mlx.py + kernels.py
Configuration	config.json
Weights	model.safetensors
Tokenizer	BPE, vocab 16,385; pad 0, EOS 1, BOS 3, mask 16,384
Training hardware	Apple M5 Max

The repository does not publish the complete training command, environment lockfile, or processed-data hashes. Reproduction is therefore architectural rather than bit-for-bit.

Limitations

• Experimental checkpoint with substantially lower TitleBench overlap scores than NeroT-29M.
• Custom MLX code is required; generic Transformers loading is not implemented despite the repository's historical library_name metadata.
• The data is English-dominant even though some underlying LMSYS conversations may be multilingual.
• Synthetic targets inherit the teacher model's style and errors.
• The benchmark guards against direct normalized-input overlap, but that does not prove absence of all semantic near-duplicates or upstream pretraining exposure.

Hardware and framework

• Trained from scratch with MLX on an Apple M5 Max.
• Custom encoder-decoder implementation included in the repository.
• Teacher attribution: BananaMind/BananaMind-Title-1.0, based on LiquidAI's LFM2.5-350M under the LFM Open License v1.0.

Files

File	Description
model.safetensors	Released model weights
config.json	Exact released architecture configuration
modeling_titan_mlx.py	MLX model implementation
kernels.py	Supporting MLX kernels
tokenizer.json	BPE tokenizer
README.md	This model card

Citation

@misc{nerot86mexp,
  author       = {j0no12},
  title        = {NeroT-86M-Exp: An Experimental Encoder-Decoder for Chat Titles},
  year         = {2026},
  howpublished = {\url{https://huggingface.co/j0no12/NeroT-86M-Exp}}
}

License

CC-BY-4.0 for the original model weights and repository material. Attribution is required. Gated LMSYS data was used locally under its own terms and is not redistributed as raw conversation text here. Upstream data and teacher-model terms remain applicable.