hoping to add multi-language support
#2
by TIN-HF - opened
After some quick tests, English conversation and coding have better performance, but using Chinese for conversation is significantly slower than the baseline.
Hi @TIN-HF , we trained this speculator on Magpie & UltraChat datasets, so we don't expect good performance on other languages, particularly with the reduced vocab likely removing important characters for other languages.
At this stage, we're not targeting multi-language support, but I will point you to the https://github.com/vllm-project/speculators library this was trained with, which you could use to add support yourself.
