gemma-3-270m-uzen-lora

This model is a fine-tuned version of davron04/gemma-3-270m-uzen-base on an unknown dataset. It achieves the following results on the evaluation set:

  • Loss: 1.6588
  • Perplexity: 5.2994

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

  • learning_rate: 2e-05
  • train_batch_size: 4
  • eval_batch_size: 8
  • seed: 42
  • distributed_type: multi-GPU
  • num_devices: 2
  • gradient_accumulation_steps: 32
  • total_train_batch_size: 256
  • total_eval_batch_size: 16
  • optimizer: Use OptimizerNames.ADAMW_TORCH with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
  • lr_scheduler_type: inverse_sqrt
  • lr_scheduler_warmup_steps: 0.01
  • num_epochs: 1
  • mixed_precision_training: Native AMP

Training results

Training Loss Epoch Step Validation Loss Perplexity
55.1648 0.1002 324 1.8270 6.2691
52.5146 0.2005 648 1.7671 5.9057
51.4328 0.3007 972 1.7364 5.7278
50.8506 0.4010 1296 1.7154 5.6088
50.1412 0.5012 1620 1.6994 5.5198
49.2935 0.6015 1944 1.6864 5.4489
49.4868 0.7017 2268 1.6757 5.3909
48.8020 0.8020 2592 1.6668 5.3422
48.1943 0.9022 2916 1.6588 5.2994

Framework versions

  • PEFT 0.18.1
  • Transformers 5.0.0
  • Pytorch 2.10.0+cu128
  • Datasets 4.8.5
  • Tokenizers 0.22.2
Downloads last month
2
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for davron04/gemma-3-270m-uzen-lora

Unable to build the model tree, the base model loops to the model itself. Learn more.