Wow
is very good. which datasets did you use for finetuning? if its not a secret
Well, according to the tags above, the datasets are HuggingFaceFW/fineweb-edu and HuggingFaceFW/dclm_100BT-shuffled
Finetuning, not pretraining
this is a Instruct, and HuggingFaceFW/dclm_100BT-shuffled, HuggingFaceFW/fineweb-edu no instruct, is text dataset, the 100m-Base is pretrain, and this is a instruct version
He was being literal. it was just /s /lh
Scroll down in the model card. There's a table.
Supervised Finetuning Data
Source Approx. share
smol-smoltalk 77.5%
Synthethic Basic Arithmetic 9.3%
qwedsacf/grade-school-math-instructions 4.5%
no_robots 3.4%
Style Rewrite of smol-smoltalk 2.5%
Style Rewrite of no_robots 1.5%
Templated b-mc2/wikihow_lists 1.2%
so, what was used to create the Instruct version from the Base model?
HuggingFaceTB/smol-smoltalk?
so, what was used to create the Instruct version from the Base model?
See my comment
okay
bruh, like what else do we have for finetuning😂
Also, I don't recommend saturating a model. i am not going to straight up spill out my techniques, but try to keep the instruct model low profile and the min intelligence and lignment shoul come from the RL
What hyper parameters did you guys use for finetuning?
depends on the model
nvm, i extracted the training_args.bin with help of gemini
here they are
training_args = TrainingArguments(
output_dir="./Supra2-100M-SFT",
run_name="Supra2-100M-SFT",
do_train=True,
do_eval=True,
eval_strategy="steps",
eval_steps=250,
per_device_train_batch_size=4,
per_device_eval_batch_size=4,
gradient_accumulation_steps=16,
num_train_epochs=1,
learning_rate=4e-05,
lr_scheduler_type="cosine",
warmup_ratio=0.03,
weight_decay=0.01,
optim="adamw_torch_fused",
adam_beta1=0.9,
adam_beta2=0.95,
bf16=True,
torch_compile=True,
dataloader_num_workers=4,
average_tokens_across_devices=True,
save_strategy="steps",
save_steps=250,
save_total_limit=3,
load_best_model_at_end=True,
metric_for_best_model="eval_loss",
greater_is_better=False,
prediction_loss_only=True,
logging_strategy="steps",
logging_steps=20,
remove_unused_columns=False,
seed=1234,
)
nvm, i extracted the training_args.bin with help of gemini
here they aretraining_args = TrainingArguments( output_dir="./Supra2-100M-SFT", run_name="Supra2-100M-SFT", do_train=True, do_eval=True, eval_strategy="steps", eval_steps=250, per_device_train_batch_size=4, per_device_eval_batch_size=4, gradient_accumulation_steps=16, num_train_epochs=1, learning_rate=4e-05, lr_scheduler_type="cosine", warmup_ratio=0.03, weight_decay=0.01, optim="adamw_torch_fused", adam_beta1=0.9, adam_beta2=0.95, bf16=True, torch_compile=True, dataloader_num_workers=4, average_tokens_across_devices=True, save_strategy="steps", save_steps=250, save_total_limit=3, load_best_model_at_end=True, metric_for_best_model="eval_loss", greater_is_better=False, prediction_loss_only=True, logging_strategy="steps", logging_steps=20, remove_unused_columns=False, seed=1234, )
This is not correct!
Gemini is dumb, even Qwen 3.8 outpeforms it
Wow, not shot Sherlock Holmes 😭
Gemini is dumb, even Qwen 3.8 outpeforms it
Google: 😔we cant even beat glm
and the audacity to make it closed source.
I mean, I get it that they put a lot of money and research into it, but imagine falling behind all of its competition, and then releasing a closed-source model...
besides multilingual work, I dont think a ingle enterprise or even solo devs use it for any meaningful work haha
Gemini is dumb, even Qwen 3.8 outpeforms it
And yet Gemini is the only one i actually use. Maybe for someone who is actually using these things for coding it's bad, but for simple information search and other basic use cases it is the most convenient option
Ok, that's fair though the hallucination rate is high
i would use claude more than gemini tbh