CodeSoft PRO

I could probably have qwen 3.8 27b (or ling 3.0 tiny for speed) classify the arena data i have (around 700 battles) which would probably be enough to train a basic classifier. Letting people pick the category might be interesting for collecting training data though.

Another suggestion - and this is just a suggestion. Allow the user to select an optional category field. Default to no category. I suggest these:

• Safety
• History
• Math
• Science
• Programming
• Judgement
• Reasoning
• Culture
• Summation
• Other

Then in the leader boards, optionally allow to pick a category, defaulting to none which would render a graph including all battles.

I was almost thinking of training a small classifier model to automatically categorize the prompt, but this might be another option

Thanks for all the suggestions! Good luck with your future models!

Haha, I’ve actually been following AI/ML developments for a year or two, and I only got seriously interested in the SLM space a couple months ago. I really only started building models myself about a month ago with MetaDiffusion though, so I’m still learning a ton and figuring things out as I go.

Thanks! I’m still only about a week into developing SLMs, so I still have a lot to learn.

That’s really generous, thank you! I think I’d rather keep the training on my end, though. I like being able to run everything myself, and I’d feel bad putting any pressure on you to spend your hardware and time on it. I really appreciate the offer!

That’s totally okay! I’m still looking for some more datasets to see where I can improve the model. I’m pretty compute and time-limited, so I usually can’t train for more than 2–3 billion tokens at this model size.

Yeah, my main concern with the benchmarks is that Sorbet-v2 is generally trailing other models in the same parameter range. For example, it scores below Supra2 Medium (25M), BananaMind 2 Mini (25M), and Veyra2 Mango (15M) on most of the benchmarks I’ve tested, although it barely passes them on HellaSwag. On ARC Easy, for example, it’s around 15% behind Supra2 Medium.

That said, you might be right that I’m simply hitting the limit of what I can squeeze out of a ~25M-parameter dense Transformer. That’s actually part of why I’m interested in trying more unusual architectures rather than just making the model bigger.