GGUF

20.4m params

#1
by CryptoAIM - opened

Is it just the standard Huggingface misscount or why is it not 300 smth billion parameters?

TeichAI org

Hey @CryptoAIM

This release only includes the Lora adapter, a tiny part of the model that we modified. We did not merge the Lora adapter into the base model for both size and convenience reasons.
The adapter can be applied to the base model without merging it into the weights beforehand.

Regards
Liontix

TeichAI org

Is it just the standard Huggingface misscount or why is it not 300 smth billion parameters?

It is a lora you load during inference. Uploaded as a lora rather than us merging the 40mb lora with the bf16 base model then uploading 1tb or more of merged quants which would be very wasteful and use a TON of storage. Loading the lora is the same effect as if when you use a image model and load a lora for a specific effect during inference rather than merge it with the image model then inference. The adapter is 40mb so the VRAM use is negligable.

Ok, thx

CompactAI changed discussion status to closed

Sign up or log in to comment