Purpose?

#2
by sorryhyun - opened

Hi, I'm quite interested in your work. So what would be the final purpose of this work? I expect enabling fp16 training for both accuracy and stability if being ambitious. Thanks for the nice work anw

It try to provide a MOSS-image-base that is easier to finetune, and some community member already report that with healed version the finetune is way more stable and effective.
It also allow FP16 to finetune/inference the model without special handling as well.

Basically it is trained to fix the bad numerical behavior in MOSS-image models, while try to align with the original output.

I will also use similar infra to train KohakuMOSS in future I guess

you think it would help with lora training? Moss always kind of freaks out after a while with crazy anatomy and stuff. (using ai toolkit).
It's by far the most b*tchiest model i ever lora trained on.
(ok, sdxl was a a challenge too)

you think it would help with lora training? Moss always kind of freaks out after a while with crazy anatomy and stuff. (using ai toolkit).
It's by far the most b*tchiest model i ever lora trained on.
(ok, sdxl was a a challenge too)

Not just "I think", but community already fully confirmed that use MOSS-image-heal help a lot in finetuning bcuz it have way more better numeric property.
Also you may want to try my trainer;https://github.com/KohakuBlueleaf/KohakuMossTrainer

i'm using moss only occasionally. So all in one place is the big benefit of aitoolkit. and fixing the few things i don't like about it (UI stuff, proper offline mode) is easy.

Sign up or log in to comment