NeroT-29M

A 29M-parameter encoder-decoder Transformer for chat title generation, trained from scratch on Apple Silicon with MLX.

Quick start (MLX)

pip install mlx mlx-lm transformers

from mlx_lm import load, generate

model, tokenizer = load("j0no12/NeroT-29M")

def generate_title(conversation: str) -> str:
    return generate(
        model,
        tokenizer,
        prompt=conversation[:2000],
        max_tokens=32,
        temp=0.8,
        top_k=50,
        repetition_penalty=1.2,
    ).strip()

conversation = """User: How do I organize a weekend trip to Boston?
Assistant: Start with dates, budget, transport, and the places you most want to visit."""
print(generate_title(conversation))

This repository uses a custom encoder-decoder configuration. If your installed mlx-lm release cannot load it directly, use the architecture notes in MODEL_INFO.md with the supplied weights and tokenizer.

Results

TitleBench (21,448 examples)

Model	Params	ROUGE-1	chrF	EM	Repetition
SupraLabs/supra-title-50m-pre	50M	0.328	0.232	3.4%	2.4%
j0no12/NeroT-29M	29M	0.296	0.201	8.9%	2.7%
SupraLabs/Supra-50M-Instruct	50M	0.057	0.042	0.8%	32.6%
BananaMind/BananaMind-Title-1.0	354M	0.471	0.345	9.3%	1.8%

All rows use the same held-out j0no12/TitleBench split. Decoding was not identical across models: NeroT used temperature 0.8, top-k 50, repetition penalty 1.2, and a 32-token limit; competitors used their documented prompts and greedy decoding. Treat the table as a practical comparison, not a decoding-controlled architecture ablation.

Takeaway: NeroT-29M is competitive for its size. It reaches 8.9% exact match—close to the 354M BananaMind model's 9.3%—while substantially outperforming the 50M general-purpose baseline on every reported metric. It does not outperform BananaMind-Title-1.0 overall.

Architecture

Component	Configuration
Model type	Encoder-decoder Transformer
Encoder / decoder layers	6 / 6
Hidden size	448
Attention heads	7 × 64-dimensional heads
FFN size	1,792
Tokenizer	t5-small tokenizer files supplied in this repository
Precision	BF16 weights; FP32 optimizer state during training

Training story

1. Base training

• Five epochs over 429,000 cleaned, deduplicated title examples.
• Recorded corpus volume: 53.9M input tokens and 3.5M target tokens per epoch.
• AdamW with peak learning rate 3e-4, 60 warmup steps, and cosine decay.
• 2.65 hours on an Apple M5 Max with 48GB unified memory.

The earlier card labeled this phase “44M tokens,” which conflicts with its own per-epoch token accounting. This card preserves the reproducible epoch and corpus figures and does not repeat that ambiguous total.

2. Continued pre-training

• Eight epochs with an additional 10M-token FineWeb-Edu sample.
• Batch mixture: 80% title data and 20% general text.
• General text objective: T5-style span corruption with a 15% mask rate.
• Length-bucketed batching delivered a reported 3.2× speedup over fixed 512-token padding.
• Validation loss improved from 4.24 to a best recorded 3.15 over 3.7 hours.

3. Decoding adjustment

A repetition penalty of 1.2 reduced repetition from 14% to 2.7% in the recorded decoding ablation. That ablation reported ROUGE-1 moving from 0.326 to 0.335; it is separate from the full-split release result of 0.296 shown above.

Data lineage

Source	Role	Rows / volume	Source terms reported by the source card
spitfire4794/chat-titles-245k	Base training	159,849	CC-BY-4.0 / ODC-BY
Michionlion/chat-titles-english	Base training	107,677	CC-BY-4.0
BananaMind/Chat-Title-200K	Base training	161,487	ID-only reconstruction from gated LMSYS data
HuggingFaceFW/fineweb-edu	Continued pre-training	~23K chunks / ~10M tokens	ODC-BY

The evaluation split contains 13,324 public-text rows and 8,124 LMSYS-derived ID-only rows. See j0no12/TitleBench for reconstruction details.

Evaluation details

• Dataset: j0no12/TitleBench, full 21,448-example split.
• Reported NeroT decoding: temperature 0.8, top-k 50, repetition penalty 1.2, maximum 32 generated tokens.
• Metrics: ROUGE-1/2/L, BLEU-4, chrF, normalized exact match, and token repetition rate.
• Full released NeroT metrics are recorded in the Hub model-index metadata above.

Reproducibility

Item	Value
Model repository revision documented by this card update	58c0cc90d6f59c7ffa65463003487ce987ac1a0b
Evaluation dataset	j0no12/TitleBench
Framework	MLX 0.32
Training hardware	Apple M5 Max, 40 GPU cores, 48GB unified memory
Runtime recorded during training	Python 3.14 (Homebrew)

The repository does not currently publish the complete training script, exact environment lockfile, or a cryptographic manifest for every training shard. Those omissions limit bit-for-bit reproduction.

Limitations

• This is a narrow title-generation model, not a general assistant.
• The benchmark comparison mixes decoding policies, so small cross-model differences should not be treated as controlled evidence.
• Training data is English-dominant and includes synthetic titles, which can transfer teacher style and errors.
• Exact match rewards one reference wording and understates valid alternative titles.
• The custom encoder-decoder implementation may require repository-specific loading code rather than generic AutoModel support.

Files

File	Description
model.safetensors	Released BF16 model weights
config.json	Encoder-decoder configuration
tokenizer.json, tokenizer_config.json	Tokenizer assets
MODEL_INFO.md	Architecture and training notes
README.md	This model card

Citation

@misc{nerot29m,
  author       = {j0no12},
  title        = {NeroT-29M: A Small Encoder-Decoder for Chat Title Generation},
  year         = {2025},
  howpublished = {\url{https://huggingface.co/j0no12/NeroT-29M}},
  note         = {Trained from scratch on Apple M5 Max with MLX}
}

License

CC-BY-4.0. Attribution is required. Upstream dataset terms remain applicable to their respective material.