20.4m params
Is it just the standard Huggingface misscount or why is it not 300 smth billion parameters?
Hey @CryptoAIM
This release only includes the Lora adapter, a tiny part of the model that we modified. We did not merge the Lora adapter into the base model for both size and convenience reasons.
The adapter can be applied to the base model without merging it into the weights beforehand.
Regards
Liontix
Is it just the standard Huggingface misscount or why is it not 300 smth billion parameters?
It is a lora you load during inference. Uploaded as a lora rather than us merging the 40mb lora with the bf16 base model then uploading 1tb or more of merged quants which would be very wasteful and use a TON of storage. Loading the lora is the same effect as if when you use a image model and load a lora for a specific effect during inference rather than merge it with the image model then inference. The adapter is 40mb so the VRAM use is negligable.
Ok, thx