Duplicate from Ihor/gliner-biomed-small-v1.0

Co-authored-by: Stepanov <Ihor@users.noreply.huggingface.co>

.gitattributes

@@ -0,0 +1,35 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,155 @@
+---
+base_model:
+- microsoft/deberta-v3-small
+datasets:
+- anthonyyazdaniml/gliner-biomed-pre-training
+- anthonyyazdaniml/gliner-biomed-post-training
+language:
+- en
+library_name: gliner
+license: apache-2.0
+metrics:
+- f1
+pipeline_tag: token-classification
+tags:
+- NER
+- GLiNER
+- information extraction
+- encoder
+- entity recognition
+- biomed
+---
+
+# GLiNER-BioMed
+
+The model was presented in the paper [GLiNER-BioMed: A Suite of Efficient Models for Open Biomedical Named Entity Recognition](https://arxiv.org/abs/2504.00676).
+
+The code is available at [https://github.com/ds4dh/GLiNER-biomed](https://github.com/ds4dh/GLiNER-biomed).
+
+**GLiNER** is a Named Entity Recognition (NER) model capable of identifying any entity type using a bidirectional transformer encoders (BERT-like). It provides a practical alternative to traditional NER models, which are limited to predefined entities, and Large Language Models (LLMs) that, despite their flexibility, are costly and large for resource-constrained scenarios.
+
+
+**GLiNER-biomed**, developed in collaboration with [DS4DH](https://www.unige.ch/medecine/radio/en/research-groups/1035teodoro) from the University of Geneva, introduces a specialized suite of efficient open biomedical NER models based on the GLiNER framework. GLiNER-biomed leverages synthetic annotations distilled from large generative biomedical language models to achieve state-of-the-art zero-shot and few-shot performance in biomedical entity recognition tasks.
+
+### How to Use
+Install the official GLiNER library with pip:
+```bash
+pip install gliner -U
+```
+
+After installing the GLiNER library, you can easily load a GLiNER-biomed model and perform named entity recognition:
+
+```python
+from gliner import GLiNER
+
+model = GLiNER.from_pretrained("Ihor/gliner-biomed-small-v1.0")
+
+text = """
+The patient, a 45-year-old male, was diagnosed with type 2 diabetes mellitus and hypertension.
+He was prescribed Metformin 500mg twice daily and Lisinopril 10mg once daily. 
+A recent lab test showed elevated HbA1c levels at 8.2%.
+"""
+
+labels = ["Disease", "Drug", "Drug dosage", "Drug frequency", "Lab test", "Lab test value", "Demographic information"]
+
+entities = model.predict_entities(text, labels, threshold=0.5)
+
+for entity in entities:
+    print(entity["text"], "=>", entity["label"])
+```
+
+Expected output:
+```
+45-year-old male => Demographic information
+type 2 diabetes mellitus => Disease
+hypertension => Disease
+Metformin => Drug
+500mg => Drug dosage
+twice daily => Drug frequency
+Lisinopril => Drug
+10mg => Drug dosage
+once daily => Drug frequency
+HbA1c levels => Lab test
+8.2% => Lab test value
+```
+
+### Benchmarking
+
+We examined our models on 8 complex real-world datasets and compared them with other GLiNER models.
+
+| Model               	| F1-score | Macro mean F1 | Macro median F1 | Weighted F1 |
+|------------------------|----------|----------------|------------------|-------------|
+| **Large models**   	|      	|            	|              	|         	|
+| [NuNER Zero](https://huggingface.co/numind/NuNER_Zero)         	| 40.87	| 21.79      	| 13.94        	| 33.67   	|
+| [NuNER Zero span](https://huggingface.co/numind/NuNER_Zero-span)    	| 40.26	| 22.51      	| 14.27        	| 32.52   	|
+| [GLiNER bio v0.1](https://huggingface.co/urchade/gliner_large_bio-v0.1)    	| 42.34	| 27.10      	| 24.44        	| 38.32   	|
+| [GLiNER bio v0.2](https://huggingface.co/urchade/gliner_large_bio-v0.2)    	| 38.66	| 25.36      	| 17.02        	| 32.42   	|
+| [GLiNER v1.0](https://huggingface.co/urchade/gliner_large-v1)        	| 47.77	| 29.60      	| 21.13        	| 40.78   	|
+| [GLiNER v2.0](https://huggingface.co/urchade/gliner_large-v2)        	| 37.38	| 21.42      	| 15.44        	| 33.11   	|
+| [GLiNER v2.1](https://huggingface.co/urchade/gliner_large-v2.1)        	| 48.04	| 29.75      	| 28.20        	| 43.43   	|
+| [GLiNER news v2.1](https://huggingface.co/EmergentMethods/gliner_large_news-v2.1)   	| 48.99	| 31.79      	| 33.77        	| 45.13   	|
+| [GLiNER v2.5](https://huggingface.co/gliner-community/gliner_large-v2.5)        	| 53.81	| 35.22      	| 35.65        	| 51.57   	|
+| **[GLiNER-biomed](https://huggingface.co/Ihor/gliner-biomed-large-v1.0)**  	| **59.77**| **40.67**  	| **42.65**    	| **58.40**   |
+| **[GLiNER-biomed-bi](https://huggingface.co/Ihor/gliner-biomed-bi-large-v1.0)**   	| 54.90	| 35.78      	| 31.66        	| 50.46   	|
+| **Base models**    	|      	|            	|              	|         	|
+| [GLiNER v1.0](https://huggingface.co/urchade/gliner_medium-v1)        	| 41.61	| 24.98      	| 10.27        	| 31.59   	|
+| [GLiNER v2.0](https://huggingface.co/urchade/gliner_medium-v2)        	| 34.33	| 24.48      	| 22.01        	| 30.58   	|
+| [GLiNER v2.1](https://huggingface.co/urchade/gliner_medium-v2.1)        	| 40.25	| 25.26      	| 14.41        	| 32.64   	|
+| [GLiNER news v2.1](https://huggingface.co/EmergentMethods/gliner_medium_news-v2.1)   	| 41.59	| 27.16      	| 17.74        	| 34.44   	|
+| [GLiNER v2.5](https://huggingface.co/gliner-community/gliner_base-v2.5)        	| 46.49	| 30.93      	| 25.26        	| 44.68   	|
+| **[GLiNER-biomed](https://huggingface.co/Ihor/gliner-biomed-base-v1.0)**  	| 54.37| **36.20**  	| **41.61**    	| 53.05   |
+| **[GLiNER-biomed-bi](https://huggingface.co/Ihor/gliner-biomed-bi-base-v1.0)**   	| **58.31**	| 35.22      	| 32.39        	| **54.91**   	|
+| **Small models**   	|      	|            	|              	|         	|
+| [GLiNER v1.0](https://huggingface.co/urchade/gliner_small-v1)        	| 40.99	| 22.81      	| 7.86         	| 31.15   	|
+| [GLiNER v2.0](https://huggingface.co/urchade/gliner_small-v2)        	| 33.55	| 21.12      	| 15.76        	| 28.78   	|
+| [GLiNER v2.1](https://huggingface.co/urchade/gliner_small-v2.1)        	| 38.45	| 23.25      	| 10.92        	| 30.67   	|
+| [GLiNER news v2.1](https://huggingface.co/EmergentMethods/gliner_small_news-v2.1)   	| 39.15	| 24.96      	| 14.48        	| 33.10   	|
+| [GLiNER v2.5](https://huggingface.co/gliner-community/gliner_small-v2.5)        	| 38.21	| 28.53      	| 18.01        	| 36.88   	|
+| **[GLiNER-biomed](https://huggingface.co/Ihor/gliner-biomed-small-v1.0)**  	| 52.53| **34.49**  	| **38.17**    	| 50.87   |
+| **[GLiNER-biomed-bi](https://huggingface.co/Ihor/gliner-biomed-bi-small-v1.0)**   	| **56.93**	| 33.88      	| 33.61        	| **53.12**   	|
+
+
+
+### Join Our Discord
+
+Connect with our community on Discord for news, support, and discussion about our models. Join [https://discord.gg/dkyeAgs9DG](https://discord.gg/dkyeAgs9DG).
+
+## Citation
+
+### This work
+If you use GLiNER-biomed models in your work, please cite:
+
+```bibtex
+@misc{yazdani2025glinerbiomedsuiteefficientmodels,
+      title={GLiNER-BioMed: A Suite of Efficient Models for Open Biomedical Named Entity Recognition}, 
+      author={Anthony Yazdani and Ihor Stepanov and Douglas Teodoro},
+      year={2025},
+      eprint={2504.00676},
+      archivePrefix={arXiv},
+      primaryClass={cs.CL},
+      url={https://arxiv.org/abs/2504.00676}, 
+}
+```
+
+### Previous work
+```bibtex
+@misc{zaratiana2023gliner,
+      title={GLiNER: Generalist Model for Named Entity Recognition using Bidirectional Transformer},
+      author={Urchade Zaratiana and Nadi Tomeh and Pierre Holat and Thierry Charnois},
+      year={2023},
+      eprint={2311.08526},
+      archivePrefix={arXiv},
+      primaryClass={cs.CL}
+}
+```
+
+```bibtex
+@misc{stepanov2024gliner,
+      title={GLiNER multi-task: Generalist Lightweight Model for Various Information Extraction Tasks},
+      author={Ihor Stepanov and Mykhailo Shtopko},
+      year={2024},
+      eprint={2406.12925},
+      archivePrefix={arXiv},
+      primaryClass={id='cs.LG' full_name='Machine Learning' is_active=True alt_name=None in_archive='cs' is_general=False description='Papers on all aspects of machine learning research (supervised, unsupervised, reinforcement learning, bandit problems, and so on) including also robustness, explanation, fairness, and methodology. cs.LG is also an appropriate primary category for applications of machine learning methods.'}
+}
+```

added_tokens.json

@@ -0,0 +1,5 @@
+{
+  "<<ENT>>": 128001,
+  "<<SEP>>": 128002,
+  "[MASK]": 128000
+}

gliner_config.json

@@ -0,0 +1,144 @@
+{
+  "class_token_index": 128001,
+  "dropout": 0.3,
+  "embed_ent_token": true,
+  "encoder_config": {
+    "_attn_implementation_autoset": false,
+    "_name_or_path": "microsoft/deberta-v3-small",
+    "add_cross_attention": false,
+    "architectures": null,
+    "attention_probs_dropout_prob": 0.1,
+    "bad_words_ids": null,
+    "begin_suppress_tokens": null,
+    "bos_token_id": null,
+    "chunk_size_feed_forward": 0,
+    "cross_attention_hidden_size": null,
+    "decoder_start_token_id": null,
+    "diversity_penalty": 0.0,
+    "do_sample": false,
+    "early_stopping": false,
+    "encoder_no_repeat_ngram_size": 0,
+    "eos_token_id": null,
+    "exponential_decay_length_penalty": null,
+    "finetuning_task": null,
+    "forced_bos_token_id": null,
+    "forced_eos_token_id": null,
+    "hidden_act": "gelu",
+    "hidden_dropout_prob": 0.1,
+    "hidden_size": 768,
+    "id2label": {
+      "0": "LABEL_0",
+      "1": "LABEL_1"
+    },
+    "initializer_range": 0.02,
+    "intermediate_size": 3072,
+    "is_decoder": false,
+    "is_encoder_decoder": false,
+    "label2id": {
+      "LABEL_0": 0,
+      "LABEL_1": 1
+    },
+    "layer_norm_eps": 1e-07,
+    "legacy": true,
+    "length_penalty": 1.0,
+    "max_length": 20,
+    "max_position_embeddings": 512,
+    "max_relative_positions": -1,
+    "min_length": 0,
+    "model_type": "deberta-v2",
+    "no_repeat_ngram_size": 0,
+    "norm_rel_ebd": "layer_norm",
+    "num_attention_heads": 12,
+    "num_beam_groups": 1,
+    "num_beams": 1,
+    "num_hidden_layers": 6,
+    "num_return_sequences": 1,
+    "output_attentions": false,
+    "output_hidden_states": false,
+    "output_scores": false,
+    "pad_token_id": 0,
+    "pooler_dropout": 0,
+    "pooler_hidden_act": "gelu",
+    "pooler_hidden_size": 768,
+    "pos_att_type": [
+      "p2c",
+      "c2p"
+    ],
+    "position_biased_input": false,
+    "position_buckets": 256,
+    "prefix": null,
+    "problem_type": null,
+    "pruned_heads": {},
+    "relative_attention": true,
+    "remove_invalid_values": false,
+    "repetition_penalty": 1.0,
+    "return_dict": true,
+    "return_dict_in_generate": false,
+    "sep_token_id": null,
+    "share_att_key": true,
+    "suppress_tokens": null,
+    "task_specific_params": null,
+    "temperature": 1.0,
+    "tf_legacy_loss": false,
+    "tie_encoder_decoder": false,
+    "tie_word_embeddings": true,
+    "tokenizer_class": null,
+    "top_k": 50,
+    "top_p": 1.0,
+    "torch_dtype": null,
+    "torchscript": false,
+    "type_vocab_size": 0,
+    "typical_p": 1.0,
+    "use_bfloat16": false,
+    "vocab_size": 128003
+  },
+  "ent_token": "<<ENT>>",
+  "eval_every": 10000,
+  "fine_tune": true,
+  "freeze_token_rep": false,
+  "fuse_layers": false,
+  "has_rnn": true,
+  "hidden_size": 512,
+  "label_smoothing": 0,
+  "labels_encoder": null,
+  "labels_encoder_config": null,
+  "labels_fusion_schema": "",
+  "log_dir": "models/",
+  "loss_alpha": 0.75,
+  "loss_gamma": 2,
+  "loss_reduction": "sum",
+  "lr_encoder": "1e-5",
+  "lr_others": "5e-5",
+  "max_grad_norm": 1.0,
+  "max_len": 2048,
+  "max_neg_type_ratio": 1,
+  "max_types": 30,
+  "max_width": 12,
+  "model_name": "microsoft/deberta-v3-small",
+  "model_type": "gliner",
+  "name": "span level gliner",
+  "num_post_fusion_layers": 1,
+  "num_rnns": 1,
+  "num_steps": 20000,
+  "post_fusion_schema": "",
+  "pre_fusion": false,
+  "prev_path": null,
+  "random_drop": true,
+  "root_dir": "gliner_logs",
+  "save_total_limit": 3,
+  "scheduler_type": "cosine",
+  "sep_token": "<<SEP>>",
+  "shuffle_types": true,
+  "size_sup": -1,
+  "span_mode": "markerV0",
+  "subtoken_pooling": "first",
+  "train_batch_size": 8,
+  "train_data": "data/ner_batches_8b_tuned_100k.json",
+  "transformers_version": "4.48.0.dev0",
+  "val_data_dir": "none",
+  "vocab_size": 128003,
+  "warmup_ratio": 0.1,
+  "weight_decay_encoder": 0.01,
+  "weight_decay_other": 0.01,
+  "words_splitter_type": "whitespace"
+}

pytorch_model.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:888b5212e6c4a66d6796d7872e8527fdecf26f296c04f3509544ed60d934652c
+size 610636974

rng_state.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6e1c4af39d092148863bc09af622ccb57c93bce265912a1f0afdde961adc6c14
+size 14244

scheduler.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e115debc88a308ec6141837d91ce27474371bbc4ba1ac15e0c651b86a69e4cd3
+size 1064

special_tokens_map.json

@@ -0,0 +1,51 @@
+{
+  "bos_token": {
+    "content": "[CLS]",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "cls_token": {
+    "content": "[CLS]",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "[SEP]",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "mask_token": {
+    "content": "[MASK]",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "[PAD]",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "sep_token": {
+    "content": "[SEP]",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "unk_token": {
+    "content": "[UNK]",
+    "lstrip": false,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  }
+}

spm.model

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c679fbf93643d19aab7ee10c0b99e460bdbc02fedf34b92b05af343b4af586fd
+size 2464616

tokenizer_config.json

@@ -0,0 +1,75 @@
+{
+  "added_tokens_decoder": {
+    "0": {
+      "content": "[PAD]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "1": {
+      "content": "[CLS]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "2": {
+      "content": "[SEP]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "3": {
+      "content": "[UNK]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128000": {
+      "content": "[MASK]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128001": {
+      "content": "<<ENT>>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128002": {
+      "content": "<<SEP>>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "bos_token": "[CLS]",
+  "clean_up_tokenization_spaces": false,
+  "cls_token": "[CLS]",
+  "do_lower_case": false,
+  "eos_token": "[SEP]",
+  "extra_special_tokens": {},
+  "mask_token": "[MASK]",
+  "model_max_length": 1000000000000000019884624838656,
+  "pad_token": "[PAD]",
+  "sep_token": "[SEP]",
+  "sp_model_kwargs": {},
+  "split_by_punct": false,
+  "tokenizer_class": "DebertaV2Tokenizer",
+  "unk_token": "[UNK]",
+  "vocab_type": "spm"
+}

trainer_state.json

@@ -0,0 +1,741 @@
+{
+  "best_metric": null,
+  "best_model_checkpoint": null,
+  "epoch": 1.7711654268508679,
+  "eval_steps": 500,
+  "global_step": 10000,
+  "is_hyper_param_search": false,
+  "is_local_process_zero": true,
+  "is_world_process_zero": true,
+  "log_history": [
+    {
+      "epoch": 0.017711654268508677,
+      "grad_norm": 282.9578857421875,
+      "learning_rate": 1.0000000000000002e-06,
+      "loss": 515.0844,
+      "step": 100
+    },
+    {
+      "epoch": 0.035423308537017355,
+      "grad_norm": 952.3278198242188,
+      "learning_rate": 2.0000000000000003e-06,
+      "loss": 282.6205,
+      "step": 200
+    },
+    {
+      "epoch": 0.053134962805526036,
+      "grad_norm": 421.5475158691406,
+      "learning_rate": 3e-06,
+      "loss": 187.263,
+      "step": 300
+    },
+    {
+      "epoch": 0.07084661707403471,
+      "grad_norm": 697.6055908203125,
+      "learning_rate": 4.000000000000001e-06,
+      "loss": 181.7124,
+      "step": 400
+    },
+    {
+      "epoch": 0.08855827134254339,
+      "grad_norm": 12816.384765625,
+      "learning_rate": 5e-06,
+      "loss": 181.1332,
+      "step": 500
+    },
+    {
+      "epoch": 0.10626992561105207,
+      "grad_norm": 943.728515625,
+      "learning_rate": 6e-06,
+      "loss": 119.6682,
+      "step": 600
+    },
+    {
+      "epoch": 0.12398157987956075,
+      "grad_norm": 1511.4503173828125,
+      "learning_rate": 7e-06,
+      "loss": 265.4774,
+      "step": 700
+    },
+    {
+      "epoch": 0.14169323414806942,
+      "grad_norm": 738.5520629882812,
+      "learning_rate": 8.000000000000001e-06,
+      "loss": 293.2895,
+      "step": 800
+    },
+    {
+      "epoch": 0.1594048884165781,
+      "grad_norm": 683.668212890625,
+      "learning_rate": 9e-06,
+      "loss": 170.2527,
+      "step": 900
+    },
+    {
+      "epoch": 0.17711654268508678,
+      "grad_norm": 2968.679443359375,
+      "learning_rate": 1e-05,
+      "loss": 307.2464,
+      "step": 1000
+    },
+    {
+      "epoch": 0.19482819695359546,
+      "grad_norm": 358.26092529296875,
+      "learning_rate": 9.99695413509548e-06,
+      "loss": 191.1959,
+      "step": 1100
+    },
+    {
+      "epoch": 0.21253985122210414,
+      "grad_norm": 814.0306396484375,
+      "learning_rate": 9.987820251299121e-06,
+      "loss": 118.1777,
+      "step": 1200
+    },
+    {
+      "epoch": 0.23025150549061282,
+      "grad_norm": 1002.783935546875,
+      "learning_rate": 9.972609476841368e-06,
+      "loss": 152.1576,
+      "step": 1300
+    },
+    {
+      "epoch": 0.2479631597591215,
+      "grad_norm": 3436.5205078125,
+      "learning_rate": 9.951340343707852e-06,
+      "loss": 139.2181,
+      "step": 1400
+    },
+    {
+      "epoch": 0.26567481402763016,
+      "grad_norm": 895.021484375,
+      "learning_rate": 9.924038765061042e-06,
+      "loss": 254.7134,
+      "step": 1500
+    },
+    {
+      "epoch": 0.28338646829613884,
+      "grad_norm": 485.51300048828125,
+      "learning_rate": 9.890738003669029e-06,
+      "loss": 130.9937,
+      "step": 1600
+    },
+    {
+      "epoch": 0.3010981225646475,
+      "grad_norm": 3067.70166015625,
+      "learning_rate": 9.851478631379982e-06,
+      "loss": 130.5014,
+      "step": 1700
+    },
+    {
+      "epoch": 0.3188097768331562,
+      "grad_norm": 710.6421508789062,
+      "learning_rate": 9.806308479691595e-06,
+      "loss": 138.9038,
+      "step": 1800
+    },
+    {
+      "epoch": 0.3365214311016649,
+      "grad_norm": 791.8126831054688,
+      "learning_rate": 9.755282581475769e-06,
+      "loss": 135.9246,
+      "step": 1900
+    },
+    {
+      "epoch": 0.35423308537017356,
+      "grad_norm": 744.6799926757812,
+      "learning_rate": 9.698463103929542e-06,
+      "loss": 132.2486,
+      "step": 2000
+    },
+    {
+      "epoch": 0.37194473963868224,
+      "grad_norm": 155.03514099121094,
+      "learning_rate": 9.635919272833938e-06,
+      "loss": 90.2381,
+      "step": 2100
+    },
+    {
+      "epoch": 0.3896563939071909,
+      "grad_norm": 553.8131103515625,
+      "learning_rate": 9.567727288213005e-06,
+      "loss": 93.5386,
+      "step": 2200
+    },
+    {
+      "epoch": 0.4073680481756996,
+      "grad_norm": 778.4810791015625,
+      "learning_rate": 9.493970231495836e-06,
+      "loss": 151.3705,
+      "step": 2300
+    },
+    {
+      "epoch": 0.4250797024442083,
+      "grad_norm": 613.4912109375,
+      "learning_rate": 9.414737964294636e-06,
+      "loss": 149.8487,
+      "step": 2400
+    },
+    {
+      "epoch": 0.44279135671271697,
+      "grad_norm": 202.41319274902344,
+      "learning_rate": 9.330127018922195e-06,
+      "loss": 121.8082,
+      "step": 2500
+    },
+    {
+      "epoch": 0.46050301098122565,
+      "grad_norm": 1176.41064453125,
+      "learning_rate": 9.24024048078213e-06,
+      "loss": 92.7877,
+      "step": 2600
+    },
+    {
+      "epoch": 0.4782146652497343,
+      "grad_norm": 2264.6943359375,
+      "learning_rate": 9.145187862775208e-06,
+      "loss": 109.0094,
+      "step": 2700
+    },
+    {
+      "epoch": 0.495926319518243,
+      "grad_norm": 1426.1497802734375,
+      "learning_rate": 9.045084971874738e-06,
+      "loss": 179.5395,
+      "step": 2800
+    },
+    {
+      "epoch": 0.5136379737867517,
+      "grad_norm": 1707.4801025390625,
+      "learning_rate": 8.94005376803361e-06,
+      "loss": 97.74,
+      "step": 2900
+    },
+    {
+      "epoch": 0.5313496280552603,
+      "grad_norm": 622.0428466796875,
+      "learning_rate": 8.83022221559489e-06,
+      "loss": 69.5907,
+      "step": 3000
+    },
+    {
+      "epoch": 0.549061282323769,
+      "grad_norm": 650.7168579101562,
+      "learning_rate": 8.715724127386971e-06,
+      "loss": 101.6783,
+      "step": 3100
+    },
+    {
+      "epoch": 0.5667729365922777,
+      "grad_norm": 741.4815673828125,
+      "learning_rate": 8.596699001693257e-06,
+      "loss": 71.9482,
+      "step": 3200
+    },
+    {
+      "epoch": 0.5844845908607864,
+      "grad_norm": 232.25784301757812,
+      "learning_rate": 8.473291852294986e-06,
+      "loss": 84.9108,
+      "step": 3300
+    },
+    {
+      "epoch": 0.602196245129295,
+      "grad_norm": 1159.53662109375,
+      "learning_rate": 8.345653031794292e-06,
+      "loss": 81.3379,
+      "step": 3400
+    },
+    {
+      "epoch": 0.6199078993978038,
+      "grad_norm": 164.65911865234375,
+      "learning_rate": 8.213938048432697e-06,
+      "loss": 82.0289,
+      "step": 3500
+    },
+    {
+      "epoch": 0.6376195536663124,
+      "grad_norm": 326.4909362792969,
+      "learning_rate": 8.078307376628292e-06,
+      "loss": 66.5258,
+      "step": 3600
+    },
+    {
+      "epoch": 0.6553312079348211,
+      "grad_norm": 404.8460693359375,
+      "learning_rate": 7.938926261462366e-06,
+      "loss": 77.1912,
+      "step": 3700
+    },
+    {
+      "epoch": 0.6730428622033298,
+      "grad_norm": 544.1323852539062,
+      "learning_rate": 7.795964517353734e-06,
+      "loss": 117.8293,
+      "step": 3800
+    },
+    {
+      "epoch": 0.6907545164718385,
+      "grad_norm": 2064.937744140625,
+      "learning_rate": 7.649596321166024e-06,
+      "loss": 80.8055,
+      "step": 3900
+    },
+    {
+      "epoch": 0.7084661707403471,
+      "grad_norm": 403.4572448730469,
+      "learning_rate": 7.500000000000001e-06,
+      "loss": 96.7227,
+      "step": 4000
+    },
+    {
+      "epoch": 0.7261778250088559,
+      "grad_norm": 666.6998291015625,
+      "learning_rate": 7.347357813929455e-06,
+      "loss": 140.3281,
+      "step": 4100
+    },
+    {
+      "epoch": 0.7438894792773645,
+      "grad_norm": 1186.9765625,
+      "learning_rate": 7.191855733945388e-06,
+      "loss": 110.6341,
+      "step": 4200
+    },
+    {
+      "epoch": 0.7616011335458732,
+      "grad_norm": 416.6499938964844,
+      "learning_rate": 7.033683215379002e-06,
+      "loss": 116.5317,
+      "step": 4300
+    },
+    {
+      "epoch": 0.7793127878143818,
+      "grad_norm": 1019.04248046875,
+      "learning_rate": 6.873032967079562e-06,
+      "loss": 159.9792,
+      "step": 4400
+    },
+    {
+      "epoch": 0.7970244420828906,
+      "grad_norm": 2007.982421875,
+      "learning_rate": 6.710100716628345e-06,
+      "loss": 96.393,
+      "step": 4500
+    },
+    {
+      "epoch": 0.8147360963513992,
+      "grad_norm": 1629.4732666015625,
+      "learning_rate": 6.545084971874738e-06,
+      "loss": 134.7197,
+      "step": 4600
+    },
+    {
+      "epoch": 0.832447750619908,
+      "grad_norm": 801.6519775390625,
+      "learning_rate": 6.378186779084996e-06,
+      "loss": 117.0068,
+      "step": 4700
+    },
+    {
+      "epoch": 0.8501594048884166,
+      "grad_norm": 572.10205078125,
+      "learning_rate": 6.209609477998339e-06,
+      "loss": 76.2736,
+      "step": 4800
+    },
+    {
+      "epoch": 0.8678710591569253,
+      "grad_norm": 307.35955810546875,
+      "learning_rate": 6.039558454088796e-06,
+      "loss": 101.7736,
+      "step": 4900
+    },
+    {
+      "epoch": 0.8855827134254339,
+      "grad_norm": 356.9403076171875,
+      "learning_rate": 5.8682408883346535e-06,
+      "loss": 96.7568,
+      "step": 5000
+    },
+    {
+      "epoch": 0.9032943676939427,
+      "grad_norm": 255.03634643554688,
+      "learning_rate": 5.695865504800328e-06,
+      "loss": 118.3934,
+      "step": 5100
+    },
+    {
+      "epoch": 0.9210060219624513,
+      "grad_norm": 567.1763305664062,
+      "learning_rate": 5.522642316338268e-06,
+      "loss": 117.496,
+      "step": 5200
+    },
+    {
+      "epoch": 0.9387176762309599,
+      "grad_norm": 1406.6500244140625,
+      "learning_rate": 5.348782368720627e-06,
+      "loss": 83.3509,
+      "step": 5300
+    },
+    {
+      "epoch": 0.9564293304994687,
+      "grad_norm": 1967.540283203125,
+      "learning_rate": 5.174497483512506e-06,
+      "loss": 136.5233,
+      "step": 5400
+    },
+    {
+      "epoch": 0.9741409847679773,
+      "grad_norm": 329.2083740234375,
+      "learning_rate": 5e-06,
+      "loss": 163.8091,
+      "step": 5500
+    },
+    {
+      "epoch": 0.991852639036486,
+      "grad_norm": 1403.033935546875,
+      "learning_rate": 4.825502516487497e-06,
+      "loss": 85.3021,
+      "step": 5600
+    },
+    {
+      "epoch": 1.0,
+      "eval_loss": 233.86021423339844,
+      "eval_runtime": 45.6534,
+      "eval_samples_per_second": 54.98,
+      "eval_steps_per_second": 13.756,
+      "step": 5646
+    },
+    {
+      "epoch": 1.0095642933049946,
+      "grad_norm": 685.2989501953125,
+      "learning_rate": 4.651217631279374e-06,
+      "loss": 189.6224,
+      "step": 5700
+    },
+    {
+      "epoch": 1.0272759475735034,
+      "grad_norm": 2071.7216796875,
+      "learning_rate": 4.477357683661734e-06,
+      "loss": 93.9828,
+      "step": 5800
+    },
+    {
+      "epoch": 1.0449876018420121,
+      "grad_norm": 3089.990966796875,
+      "learning_rate": 4.304134495199675e-06,
+      "loss": 95.2696,
+      "step": 5900
+    },
+    {
+      "epoch": 1.0626992561105206,
+      "grad_norm": 318.3044738769531,
+      "learning_rate": 4.131759111665349e-06,
+      "loss": 93.2886,
+      "step": 6000
+    },
+    {
+      "epoch": 1.0804109103790294,
+      "grad_norm": 959.2510375976562,
+      "learning_rate": 3.960441545911205e-06,
+      "loss": 60.9704,
+      "step": 6100
+    },
+    {
+      "epoch": 1.098122564647538,
+      "grad_norm": 434.56304931640625,
+      "learning_rate": 3.790390522001662e-06,
+      "loss": 170.5878,
+      "step": 6200
+    },
+    {
+      "epoch": 1.1158342189160468,
+      "grad_norm": 437.1268310546875,
+      "learning_rate": 3.6218132209150047e-06,
+      "loss": 63.596,
+      "step": 6300
+    },
+    {
+      "epoch": 1.1335458731845554,
+      "grad_norm": 683.8363037109375,
+      "learning_rate": 3.4549150281252635e-06,
+      "loss": 88.3637,
+      "step": 6400
+    },
+    {
+      "epoch": 1.151257527453064,
+      "grad_norm": 1063.7440185546875,
+      "learning_rate": 3.289899283371657e-06,
+      "loss": 90.7109,
+      "step": 6500
+    },
+    {
+      "epoch": 1.1689691817215728,
+      "grad_norm": 500.3209533691406,
+      "learning_rate": 3.12696703292044e-06,
+      "loss": 106.912,
+      "step": 6600
+    },
+    {
+      "epoch": 1.1866808359900816,
+      "grad_norm": 5315.34765625,
+      "learning_rate": 2.966316784621e-06,
+      "loss": 66.9506,
+      "step": 6700
+    },
+    {
+      "epoch": 1.20439249025859,
+      "grad_norm": 390.86773681640625,
+      "learning_rate": 2.8081442660546126e-06,
+      "loss": 72.4987,
+      "step": 6800
+    },
+    {
+      "epoch": 1.2221041445270988,
+      "grad_norm": 714.1128540039062,
+      "learning_rate": 2.6526421860705474e-06,
+      "loss": 80.7786,
+      "step": 6900
+    },
+    {
+      "epoch": 1.2398157987956075,
+      "grad_norm": 625.9713745117188,
+      "learning_rate": 2.5000000000000015e-06,
+      "loss": 105.9008,
+      "step": 7000
+    },
+    {
+      "epoch": 1.257527453064116,
+      "grad_norm": 798.7731323242188,
+      "learning_rate": 2.3504036788339763e-06,
+      "loss": 110.6368,
+      "step": 7100
+    },
+    {
+      "epoch": 1.2752391073326248,
+      "grad_norm": 503.0509948730469,
+      "learning_rate": 2.204035482646267e-06,
+      "loss": 65.3358,
+      "step": 7200
+    },
+    {
+      "epoch": 1.2929507616011335,
+      "grad_norm": 416.53094482421875,
+      "learning_rate": 2.061073738537635e-06,
+      "loss": 61.5411,
+      "step": 7300
+    },
+    {
+      "epoch": 1.3106624158696423,
+      "grad_norm": 3666.154541015625,
+      "learning_rate": 1.9216926233717087e-06,
+      "loss": 84.1701,
+      "step": 7400
+    },
+    {
+      "epoch": 1.328374070138151,
+      "grad_norm": 330.82928466796875,
+      "learning_rate": 1.7860619515673034e-06,
+      "loss": 63.6358,
+      "step": 7500
+    },
+    {
+      "epoch": 1.3460857244066595,
+      "grad_norm": 331.7156066894531,
+      "learning_rate": 1.6543469682057105e-06,
+      "loss": 116.5511,
+      "step": 7600
+    },
+    {
+      "epoch": 1.3637973786751683,
+      "grad_norm": 1124.501708984375,
+      "learning_rate": 1.5267081477050132e-06,
+      "loss": 73.444,
+      "step": 7700
+    },
+    {
+      "epoch": 1.381509032943677,
+      "grad_norm": 4456.3701171875,
+      "learning_rate": 1.4033009983067454e-06,
+      "loss": 125.1651,
+      "step": 7800
+    },
+    {
+      "epoch": 1.3992206872121855,
+      "grad_norm": 254.76058959960938,
+      "learning_rate": 1.2842758726130283e-06,
+      "loss": 84.9094,
+      "step": 7900
+    },
+    {
+      "epoch": 1.4169323414806942,
+      "grad_norm": 243.16725158691406,
+      "learning_rate": 1.1697777844051105e-06,
+      "loss": 70.4103,
+      "step": 8000
+    },
+    {
+      "epoch": 1.434643995749203,
+      "grad_norm": 444.5528869628906,
+      "learning_rate": 1.0599462319663906e-06,
+      "loss": 148.3668,
+      "step": 8100
+    },
+    {
+      "epoch": 1.4523556500177117,
+      "grad_norm": 712.6991577148438,
+      "learning_rate": 9.549150281252633e-07,
+      "loss": 78.7889,
+      "step": 8200
+    },
+    {
+      "epoch": 1.4700673042862205,
+      "grad_norm": 1260.743408203125,
+      "learning_rate": 8.54812137224792e-07,
+      "loss": 133.5537,
+      "step": 8300
+    },
+    {
+      "epoch": 1.487778958554729,
+      "grad_norm": 375.3415832519531,
+      "learning_rate": 7.597595192178702e-07,
+      "loss": 96.2294,
+      "step": 8400
+    },
+    {
+      "epoch": 1.5054906128232377,
+      "grad_norm": 1678.25634765625,
+      "learning_rate": 6.698729810778065e-07,
+      "loss": 121.3787,
+      "step": 8500
+    },
+    {
+      "epoch": 1.5232022670917464,
+      "grad_norm": 844.283447265625,
+      "learning_rate": 5.852620357053651e-07,
+      "loss": 94.1799,
+      "step": 8600
+    },
+    {
+      "epoch": 1.540913921360255,
+      "grad_norm": 706.9551391601562,
+      "learning_rate": 5.06029768504166e-07,
+      "loss": 107.5293,
+      "step": 8700
+    },
+    {
+      "epoch": 1.5586255756287637,
+      "grad_norm": 1072.598388671875,
+      "learning_rate": 4.322727117869951e-07,
+      "loss": 90.6825,
+      "step": 8800
+    },
+    {
+      "epoch": 1.5763372298972724,
+      "grad_norm": 534.5012817382812,
+      "learning_rate": 3.6408072716606346e-07,
+      "loss": 99.7579,
+      "step": 8900
+    },
+    {
+      "epoch": 1.594048884165781,
+      "grad_norm": 76949.8984375,
+      "learning_rate": 3.015368960704584e-07,
+      "loss": 186.237,
+      "step": 9000
+    },
+    {
+      "epoch": 1.61176053843429,
+      "grad_norm": 294.4422302246094,
+      "learning_rate": 2.447174185242324e-07,
+      "loss": 136.7301,
+      "step": 9100
+    },
+    {
+      "epoch": 1.6294721927027984,
+      "grad_norm": 347.99017333984375,
+      "learning_rate": 1.9369152030840553e-07,
+      "loss": 103.0369,
+      "step": 9200
+    },
+    {
+      "epoch": 1.6471838469713072,
+      "grad_norm": 2896.26806640625,
+      "learning_rate": 1.4852136862001766e-07,
+      "loss": 73.6874,
+      "step": 9300
+    },
+    {
+      "epoch": 1.664895501239816,
+      "grad_norm": 398.4006042480469,
+      "learning_rate": 1.0926199633097156e-07,
+      "loss": 75.4443,
+      "step": 9400
+    },
+    {
+      "epoch": 1.6826071555083244,
+      "grad_norm": 1028.520751953125,
+      "learning_rate": 7.59612349389599e-08,
+      "loss": 75.612,
+      "step": 9500
+    },
+    {
+      "epoch": 1.7003188097768331,
+      "grad_norm": 516.2049560546875,
+      "learning_rate": 4.865965629214819e-08,
+      "loss": 94.1481,
+      "step": 9600
+    },
+    {
+      "epoch": 1.7180304640453419,
+      "grad_norm": 1118.83935546875,
+      "learning_rate": 2.7390523158633552e-08,
+      "loss": 122.414,
+      "step": 9700
+    },
+    {
+      "epoch": 1.7357421183138504,
+      "grad_norm": 372.98089599609375,
+      "learning_rate": 1.2179748700879013e-08,
+      "loss": 73.4527,
+      "step": 9800
+    },
+    {
+      "epoch": 1.7534537725823593,
+      "grad_norm": 1152.221435546875,
+      "learning_rate": 3.0458649045211897e-09,
+      "loss": 55.3135,
+      "step": 9900
+    },
+    {
+      "epoch": 1.7711654268508679,
+      "grad_norm": 729.0238037109375,
+      "learning_rate": 0.0,
+      "loss": 114.0674,
+      "step": 10000
+    }
+  ],
+  "logging_steps": 100,
+  "max_steps": 10000,
+  "num_input_tokens_seen": 0,
+  "num_train_epochs": 2,
+  "save_steps": 5000,
+  "stateful_callbacks": {
+    "TrainerControl": {
+      "args": {
+        "should_epoch_stop": false,
+        "should_evaluate": false,
+        "should_log": false,
+        "should_save": true,
+        "should_training_stop": true
+      },
+      "attributes": {}
+    }
+  },
+  "total_flos": 0.0,
+  "train_batch_size": 4,
+  "trial_name": null,
+  "trial_params": null
+}