Post
54
Super impressed with the UltraBERT team at @dmis-lab โ @affjljoo3581 , @Taewhoo and Jaewoo Kang โ for continuing to push encoders forward. Really impressive work!
I knocked up an MoE version on a single A100, entirely for fun: UltraBERT-Embed-MoE-8x.
8 experts per routed layer, 2 active per token. Works pretty well in my small retrieval tests!
Huge credit to the original team for making this possible.
Original: dmis-lab/UltraBERT-Embed-150M
MoE: takarajordan/UltraBERT-Embed-MoE-8x
I knocked up an MoE version on a single A100, entirely for fun: UltraBERT-Embed-MoE-8x.
8 experts per routed layer, 2 active per token. Works pretty well in my small retrieval tests!
Huge credit to the original team for making this possible.
Original: dmis-lab/UltraBERT-Embed-150M
MoE: takarajordan/UltraBERT-Embed-MoE-8x